Nov 01 19:59:37.120321 ip-10-0-138-177 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 19:59:37.120330 ip-10-0-138-177 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 19:59:37.120337 ip-10-0-138-177 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 19:59:37.120553 ip-10-0-138-177 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 19:59:47.369613 ip-10-0-138-177 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 19:59:47.369631 ip-10-0-138-177 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e67060a469584af08c76085d4325ee95 -- Nov 01 20:01:56.416917 ip-10-0-138-177 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:01:56.803984 ip-10-0-138-177 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:01:56.803984 ip-10-0-138-177 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:01:56.803984 ip-10-0-138-177 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:01:56.803984 ip-10-0-138-177 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:01:56.808346 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808282 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.808346 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808346 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.808346 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808350 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808353 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808356 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808359 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808362 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808365 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808368 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808371 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808373 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808377 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808379 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808382 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808385 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808387 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808390 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808393 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808396 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808399 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808401 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.808454 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808404 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808410 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808414 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808416 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808419 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808422 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808424 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808427 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808430 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808433 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808435 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808438 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808441 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808443 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808446 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808449 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808451 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808454 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.808924 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808458 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808462 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808466 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808469 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808471 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808474 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808477 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808480 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808483 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808485 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808488 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808491 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808493 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808496 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808499 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808502 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808505 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808508 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808511 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.809365 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808513 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808517 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808521 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808524 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808527 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808530 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808533 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808535 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808538 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808540 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808543 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808545 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808548 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808551 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808554 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808556 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808559 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808562 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808564 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.809816 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808567 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808570 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808573 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808575 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808578 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808580 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808583 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808585 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808588 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808590 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808594 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808596 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808599 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808602 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808605 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808607 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808611 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808613 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808616 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808619 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.810283 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808621 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808624 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808626 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808629 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808631 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808645 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808649 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808651 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.808654 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809483 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809490 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809493 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809496 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809500 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809503 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809506 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809509 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809512 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809514 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809517 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.810755 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809519 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809523 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809526 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809528 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809531 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809533 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809536 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809539 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809541 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809543 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809546 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809549 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809551 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809554 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809556 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809559 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809561 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809564 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809567 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809569 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.811263 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809572 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809574 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809577 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809579 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809582 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809584 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809588 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809594 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809597 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809599 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809601 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809604 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809606 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809609 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809611 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809614 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809616 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809618 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809621 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809623 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.811760 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809626 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809629 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809632 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809649 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809652 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809654 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809657 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809660 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809662 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809665 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809668 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809671 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809673 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809676 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809679 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809681 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809684 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809686 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809692 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.812259 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809696 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809700 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809703 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809706 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809709 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809713 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809717 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809720 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809723 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809726 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809731 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809734 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809736 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809739 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809741 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809744 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809746 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809749 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.812721 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809751 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809754 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809756 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809759 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809762 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809764 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809767 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809769 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809772 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809774 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809777 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809779 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809783 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809786 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809788 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809791 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.809794 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810317 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810322 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810325 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.813172 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810327 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810330 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810333 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810335 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810339 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810343 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810346 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810349 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810352 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810355 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810357 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810360 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810362 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810365 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810368 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810371 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810374 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810377 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810379 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.813676 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810382 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810384 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810386 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810389 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810392 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810395 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810397 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810400 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810403 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810406 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810409 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810412 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810414 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810417 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810419 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810422 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810424 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810427 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810429 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.814132 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810432 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810434 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810437 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810439 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810442 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810444 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810447 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810449 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810452 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810454 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810456 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810459 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810461 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810464 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810466 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810468 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810471 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810474 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810476 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810479 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.814560 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810482 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810484 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810494 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810497 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810499 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810502 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810505 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810508 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810510 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810513 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810515 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810517 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810520 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810523 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810526 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810531 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810533 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810536 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810539 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.815041 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810541 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810544 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810546 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810548 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810551 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810553 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810556 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810558 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810561 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810563 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810566 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810569 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810572 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810574 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810577 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810579 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810582 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810584 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810587 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810589 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.815489 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810592 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810594 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810596 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810599 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810601 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.810604 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811333 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811342 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811351 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811356 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811360 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811363 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811372 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811375 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811379 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811382 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811387 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811390 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811394 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811396 2573 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811399 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811402 2573 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811405 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:01:56.815969 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811408 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811413 2573 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811416 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811419 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811422 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811425 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811430 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811433 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811436 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811440 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811443 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811446 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811449 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811452 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811455 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811459 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811462 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811466 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811469 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811472 2573 flags.go:64] FLAG: --enable-server="true" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811475 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811479 2573 flags.go:64] FLAG: --event-burst="100" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811482 2573 flags.go:64] FLAG: --event-qps="50" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811485 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:01:56.816590 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811488 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811493 2573 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811497 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811500 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811504 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811507 2573 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811510 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811513 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811516 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811519 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811521 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811524 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811527 2573 flags.go:64] FLAG: --feature-gates="" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811531 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811534 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811540 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811543 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811546 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811549 2573 flags.go:64] FLAG: --help="false" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811552 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811555 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811558 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811561 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:01:56.817236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811565 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811568 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811573 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811576 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811579 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811582 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811585 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811588 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811591 2573 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811594 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811598 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811602 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811605 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811607 2573 flags.go:64] FLAG: --lock-file="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811610 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811613 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811616 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811622 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811624 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811628 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811630 2573 flags.go:64] FLAG: --logging-format="text" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811633 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811651 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811654 2573 flags.go:64] FLAG: --manifest-url="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811657 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:01:56.817809 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811661 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811665 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811669 2573 flags.go:64] FLAG: --max-pods="110" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811672 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811675 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811677 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811681 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811683 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811687 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811691 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811698 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811700 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811703 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811707 2573 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811710 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811712 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811715 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811719 2573 flags.go:64] FLAG: --port="10250" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811722 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811725 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ebfd44319ea487b5" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811729 2573 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811732 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811734 2573 flags.go:64] FLAG: --register-node="true" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811737 2573 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811741 2573 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811744 2573 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:01:56.818401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811747 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811750 2573 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811754 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811757 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811760 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811763 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811766 2573 flags.go:64] FLAG: --runonce="false" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811769 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811773 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811776 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811779 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811781 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811784 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811787 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811791 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811795 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811798 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811801 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811803 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811806 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811809 2573 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811812 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811817 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811821 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811824 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:01:56.819029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811828 2573 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811831 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811834 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811837 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811840 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811843 2573 flags.go:64] FLAG: --v="2" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811848 2573 flags.go:64] FLAG: --version="false" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811852 2573 flags.go:64] FLAG: --vmodule="" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811856 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.811859 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812007 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812011 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812014 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812017 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812020 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812023 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812025 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812028 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812031 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812034 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812036 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.819647 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812039 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812042 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812045 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812048 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812051 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812053 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812055 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812058 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812061 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812065 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812068 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812070 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812073 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812075 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812078 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812081 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812083 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812085 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812088 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.820130 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812090 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812093 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812095 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812098 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812100 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812103 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812106 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812108 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812111 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812114 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812117 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812120 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812122 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812125 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812129 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812132 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812134 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812137 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812139 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812142 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.820576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812144 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812147 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812152 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812155 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812157 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812160 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812162 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812165 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812167 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812170 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812172 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812175 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812177 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812180 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812184 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812188 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812191 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812194 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812196 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.821118 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812199 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812201 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812204 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812207 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812209 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812213 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812218 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812222 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812225 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812228 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812231 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812233 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812236 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812238 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812241 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812245 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812248 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812250 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812253 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.821576 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812256 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812258 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812261 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812263 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812266 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812269 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812271 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812274 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812277 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812279 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812282 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812284 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812287 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812289 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812292 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812294 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812297 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.822184 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812300 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.812326 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812360 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812366 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812370 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812373 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812376 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812378 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812381 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812383 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812386 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812388 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812392 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812395 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812398 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812400 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.822588 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812403 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812405 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812409 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812413 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812415 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812418 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812421 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812423 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812426 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812428 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812430 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812433 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812437 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812440 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812443 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812446 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812449 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812452 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.823072 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812454 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812459 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812461 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812464 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812466 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812469 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812471 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812474 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812476 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812479 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812483 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812485 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812488 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812491 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812493 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812496 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812498 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812501 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812504 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812506 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.823511 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812508 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812511 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812514 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812516 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812519 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812522 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812524 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812527 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812530 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812532 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812534 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812537 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812540 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812544 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812546 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812549 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812551 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812554 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812556 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.824012 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812559 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812561 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812564 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812568 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812570 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812573 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812576 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812578 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812581 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812583 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812586 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812588 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812591 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812593 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812596 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812598 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812601 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812603 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812606 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.824462 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812608 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812611 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812613 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812616 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812618 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812622 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812625 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812629 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812631 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812644 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812647 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812650 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812652 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812655 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812657 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.812660 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.818603 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.818620 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.818631 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:01:56.824917 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.818648 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818703 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818709 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818713 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818716 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818718 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818721 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818724 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818727 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818729 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818732 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818734 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818738 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818740 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818743 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818747 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818751 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818754 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818757 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.825380 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818760 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818762 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818765 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818768 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818770 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818773 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818776 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818779 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818781 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818784 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818787 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818791 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818793 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818796 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818798 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818801 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818804 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818807 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818809 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.825919 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818812 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818814 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818817 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818820 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818822 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818825 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818828 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818830 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818833 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818835 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818838 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818840 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818843 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818845 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818848 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818850 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818853 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818855 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818858 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818860 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.826446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818864 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818868 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818871 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818874 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818877 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818880 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818882 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818885 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818888 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818890 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818893 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818895 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818898 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818900 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818903 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818905 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818908 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818910 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818913 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.827097 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818915 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818932 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818935 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818939 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818942 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818945 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818947 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818950 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818952 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818955 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818958 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818961 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818963 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818966 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818969 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818971 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818974 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818976 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818979 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.827601 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818982 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818985 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818987 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818990 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818992 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818995 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.818998 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819000 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819002 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819005 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819007 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.819034 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819172 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819178 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819181 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819184 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.828116 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819187 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819190 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819193 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819195 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819198 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819200 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819203 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819205 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819208 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819210 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819213 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819216 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819218 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819221 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819224 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819226 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819229 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819232 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819235 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.828543 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819238 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819240 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819243 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819245 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819248 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819250 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819253 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819255 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819258 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819260 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819263 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819267 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819270 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819273 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819276 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819279 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819282 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819285 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819287 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.829087 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819290 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819292 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819295 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819297 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819300 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819302 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819305 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819307 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819310 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819313 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819315 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819318 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819321 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819323 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819326 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819329 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819331 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819334 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819336 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819339 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.829586 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819341 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819344 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819346 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819349 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819352 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819354 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819357 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819359 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819362 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819364 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819367 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819369 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819372 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819375 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819378 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819380 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819382 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819385 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819387 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819390 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.830209 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819392 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819395 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819397 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819400 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819402 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819405 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819408 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819410 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819412 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819415 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819417 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819420 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819422 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819425 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819427 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819430 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819433 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819437 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819439 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.830784 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819442 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819445 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819447 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819450 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819452 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.819475 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819518 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819522 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819525 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819527 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819530 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819533 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819536 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819538 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819541 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819543 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:01:56.831237 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819546 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819548 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819551 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819554 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819557 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819559 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819562 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819565 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819567 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819570 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819572 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819575 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819577 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819580 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819582 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819585 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819590 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819593 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819595 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:01:56.831788 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819598 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819600 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819602 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819605 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819607 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819610 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819612 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819615 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819617 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819620 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819622 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819625 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819628 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819630 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819633 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819647 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819651 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819656 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819659 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:01:56.832381 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819662 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819665 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819667 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819670 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819672 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819674 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819677 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819680 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819682 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819684 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819687 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819689 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819692 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819694 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819697 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819699 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819702 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819704 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819706 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819709 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:01:56.832848 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819711 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819714 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819717 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819721 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819724 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819727 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819732 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819734 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819737 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819740 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819743 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819745 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819748 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819750 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819753 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819755 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819758 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:01:56.833314 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819760 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819765 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819769 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819771 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819774 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819777 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819780 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819782 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819785 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819787 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819789 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819792 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819794 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819797 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819799 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819802 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819804 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819810 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:56.819812 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:01:56.833747 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.819862 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:01:56.834194 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.821787 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 20:01:56.834194 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.822709 2573 server.go:1014] "Starting client certificate rotation" Nov 01 20:01:56.834194 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.822801 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:01:56.834194 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.822845 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:01:56.834194 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.824527 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:01:56.834194 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.828567 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:01:56.838524 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.838508 2573 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:01:56.840694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.840679 2573 log.go:25] "Validated CRI v1 image API" Nov 01 20:01:56.842701 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.842623 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:01:56.844927 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.844909 2573 fs.go:135] Filesystem UUIDs: map[498a6f32-2f0b-4c65-b99d-e9484dd74add:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cb440668-8f60-45b8-8567-fce71e3e278c:/dev/nvme0n1p3] Nov 01 20:01:56.844981 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.844927 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 20:01:56.849834 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.849727 2573 manager.go:217] Machine: {Timestamp:2026-11-01 20:01:56.848426674 +0000 UTC m=+0.320635911 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098003 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af26b6daa089661be53f69db44de2 SystemUUID:ec2af26b-6daa-0896-61be-53f69db44de2 BootID:e67060a4-6958-4af0-8c76-085d4325ee95 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:39:0a:0d:7c:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:39:0a:0d:7c:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:ff:cc:cc:64:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:01:56.850248 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.850238 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:01:56.850344 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.850332 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:01:56.852973 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.852948 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:01:56.853090 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.852973 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-177.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:01:56.853135 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.853092 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:01:56.853135 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.853099 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:01:56.853135 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.853125 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:01:56.853212 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.853137 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:01:56.853836 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.853826 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:01:56.857336 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.857323 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:01:56.857451 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.857442 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:01:56.858032 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.858023 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:01:56.858068 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.858035 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:01:56.858068 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.858044 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:01:56.858068 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.858052 2573 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:01:56.858068 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.858059 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:01:56.858335 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.858324 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:01:56.860162 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.860146 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:01:56.863849 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.863836 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:01:56.864458 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864442 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:01:56.864490 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864476 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:01:56.864657 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864632 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:01:56.864694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864668 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:01:56.864694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864680 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:01:56.864694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864686 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:01:56.864694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864691 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:01:56.864694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.864696 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:01:56.865034 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865026 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:01:56.865061 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865034 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:01:56.865061 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865041 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:01:56.865061 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865047 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:01:56.865061 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865060 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:01:56.865254 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865071 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:01:56.865693 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865684 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:01:56.865722 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.865693 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:01:56.869086 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.869074 2573 server.go:1263] "Started kubelet" Nov 01 20:01:56.869271 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.869234 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:01:56.869337 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.869257 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:01:56.869398 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.869381 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:01:56.869631 ip-10-0-138-177 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:01:56.873384 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.873333 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:01:56.876254 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.876236 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:01:56.876356 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:56.876337 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 20:01:56.879169 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.879155 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:01:56.879238 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.879169 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:01:56.879658 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:56.879622 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:01:56.879963 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.879943 2573 factory.go:55] Registering systemd factory Nov 01 20:01:56.880045 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.879966 2573 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:01:56.880045 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.879941 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:01:56.880045 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.879941 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:01:56.880045 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880022 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:01:56.880230 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880112 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:01:56.880230 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880121 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:01:56.880230 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880211 2573 factory.go:153] Registering CRI-O factory Nov 01 20:01:56.880230 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880224 2573 factory.go:223] Registration of the crio container factory successfully Nov 01 20:01:56.880402 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880277 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:01:56.880402 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880295 2573 factory.go:103] Registering Raw factory Nov 01 20:01:56.880402 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.880309 2573 manager.go:1199] Started watching for new ooms in manager Nov 01 20:01:56.881031 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.881016 2573 manager.go:319] Starting recovery of all containers Nov 01 20:01:56.889051 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.889032 2573 manager.go:324] Recovery completed Nov 01 20:01:56.894583 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.894570 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:56.896204 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896179 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:56.896271 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896212 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:56.896271 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896227 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:56.896761 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896747 2573 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:01:56.896761 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896759 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:01:56.896839 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896774 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:56.896839 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.896775 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:01:56.897791 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.897778 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:56.897863 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.897801 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:56.897863 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.897810 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:56.899177 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.899052 2573 policy_none.go:50] "Start" Nov 01 20:01:56.899177 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.899068 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:01:56.899177 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.899081 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:01:56.901280 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.901268 2573 policy_none.go:44] "Start" Nov 01 20:01:56.928184 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.928159 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:01:56.930458 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.930445 2573 manager.go:341] "Starting Device Plugin manager" Nov 01 20:01:56.930514 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:56.930473 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:01:56.930514 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.930481 2573 server.go:87] "Starting device plugin registration server" Nov 01 20:01:56.930686 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.930672 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:01:56.930752 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.930689 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:01:56.930940 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.930873 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:01:56.931172 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.931086 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:01:56.931172 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.931100 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:01:56.931442 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:56.931425 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:01:56.931496 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:56.931480 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:01:56.997921 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.997902 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:01:56.999527 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.998943 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:01:56.999527 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.998964 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:01:56.999527 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:56.998983 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:01:56.999527 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:56.999018 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 20:01:57.014071 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.014051 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-177.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 20:01:57.014071 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:57.014047 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-177.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 20:01:57.014770 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:57.014037 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-177.us-west-1.compute.internal.18e404558026956e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-177.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-177.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:01:56.869043566 +0000 UTC m=+0.341252804,LastTimestamp:2026-11-01 20:01:56.869043566 +0000 UTC m=+0.341252804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-177.us-west-1.compute.internal,}" Nov 01 20:01:57.031223 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.031208 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:57.031891 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.031876 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:57.031952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.031904 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:57.031952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.031920 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:57.032035 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.031948 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.096286 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.096242 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:01:57.096353 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.096292 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hn2x" Nov 01 20:01:57.099710 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.099692 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal"] Nov 01 20:01:57.099754 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.099741 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:57.100439 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.100426 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:57.100487 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.100449 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:57.100487 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.100458 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:57.102240 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102229 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:57.102375 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.102411 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102393 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:57.102886 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102871 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:57.102952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102873 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:57.102952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102901 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:57.102952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102911 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:57.102952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102921 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:57.102952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.102932 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:57.104712 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.104699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.104754 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.104721 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:01:57.105350 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.105335 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:01:57.105401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.105367 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:01:57.105401 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.105382 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:01:57.131477 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:57.131461 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.134501 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:57.134480 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.168127 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.168106 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:57.169926 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.169913 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:57.181454 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.181438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.181511 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.181459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.181511 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.181484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f167604710ff032b6c087eabbd79a27-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"6f167604710ff032b6c087eabbd79a27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.266282 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.266255 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:57.267837 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.267819 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:57.268100 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.268083 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.279589 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.279571 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.281763 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.281748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.281816 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.281770 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.281816 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.281797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f167604710ff032b6c087eabbd79a27-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"6f167604710ff032b6c087eabbd79a27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.281894 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.281857 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.281894 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.281867 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.281959 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.281864 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f167604710ff032b6c087eabbd79a27-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"6f167604710ff032b6c087eabbd79a27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.338897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.338776 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:01:57.339534 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.339522 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.398272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.398231 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:01:57.433706 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.433683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.436818 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.436803 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:01:57.823011 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.822953 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 20:01:57.823491 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.823101 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:01:57.823491 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.823142 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:01:57.823491 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.823158 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 20:01:57.823491 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:57.823186 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a4f7fa6edf75146a881ba8b0008bd1da-8a4abc42788d6c16.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-177.us-west-1.compute.internal.18e4045581c4dfa9\": read tcp 10.0.138.177:37284->3.138.241.54:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-177.us-west-1.compute.internal.18e4045581c4dfa9 default 5158 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-177.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-177.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-177.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 20:01:56 +0000 UTC,LastTimestamp:2026-11-01 20:01:57.10043783 +0000 UTC m=+0.572647067,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-177.us-west-1.compute.internal,}" Nov 01 20:01:57.858691 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.858667 2573 apiserver.go:52] "Watching apiserver" Nov 01 20:01:57.879972 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:57.879950 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 20:01:58.100868 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.100805 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:56:57 +0000 UTC" deadline="2028-03-29 11:08:32.595384443 +0000 UTC" Nov 01 20:01:58.100868 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.100832 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12327h6m34.494555118s" Nov 01 20:01:58.134154 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.134127 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:01:58.134497 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.134475 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-9hk7d","kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-4njz9","openshift-multus/multus-ldqdw","openshift-network-diagnostics/network-check-target-h5w6p","openshift-multus/multus-additional-cni-plugins-7zvvr","openshift-multus/network-metrics-daemon-88xp6","kube-system/konnectivity-agent-c5z7h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8","openshift-dns/node-resolver-qc9m9","openshift-image-registry/node-ca-zw4q4","kube-system/global-pull-secret-syncer-tn5w8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-8wmlv"] Nov 01 20:01:58.137210 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.137185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.139436 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.139414 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.141739 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.141716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.143959 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.143934 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:01:58.144072 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.144001 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:01:58.146205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.146183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.148447 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.148429 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.148556 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.148489 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:01:58.150801 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.150780 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.153105 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.153084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.155471 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.155444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.158037 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.158017 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.160696 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.160677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.160781 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.160739 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:01:58.163056 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.163040 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.181358 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.181332 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:01:58.185920 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.185896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-k8s-cni-cncf-io\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186002 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.185931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.186002 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.185960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-log-socket\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.186083 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-slash\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.186083 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.186149 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-run\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.186149 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.186212 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-hostroot\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186264 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpb4w\" (UniqueName: \"kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.186303 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186349 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-kubelet\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186403 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186450 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.186504 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk569\" (UniqueName: \"kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.186504 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-netns\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.186504 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.186632 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-var-lib-kubelet\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.186632 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.186632 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-socket-dir-parent\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186632 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-system-cni-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.186832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.186832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:01:58.186832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-system-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.186832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.186832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-modprobe-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.187036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.187036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-etc-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-lib-modules\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.187036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186946 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-multus\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.187036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.186997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-kubelet\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-var-lib-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187067 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-systemd\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187116 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-kubelet-config\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-cnibin\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187247 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b696755-be38-4546-82e1-9b485bdef642-hosts-file\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqjw\" (UniqueName: \"kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.187339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-netd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187474 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cnibin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysconfig\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.187663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-systemd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-bin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-dbus\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-conf-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187947 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-multus-certs\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.187999 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.187976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-node-log\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-host\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188133 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96z8\" (UniqueName: \"kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-netns\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-etc-kubernetes\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188233 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcr84\" (UniqueName: \"kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-conf\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.188356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-ovn\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j2h\" (UniqueName: \"kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sd5\" (UniqueName: \"kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-device-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188474 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-host\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-bin\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-kubernetes\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-sys\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188603 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnqc\" (UniqueName: \"kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-host-slash\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcntf\" (UniqueName: \"kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.188897 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.188668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-systemd-units\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.193559 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.193538 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:01:58.194107 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.193857 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:01:58.194107 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.193947 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:01:58.194107 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.194004 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.194107 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.194068 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.195373 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.195141 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" Nov 01 20:01:58.196315 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.196295 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:01:58.198347 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198324 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.198347 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198336 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.198531 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198373 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:01:58.198531 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198382 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.198531 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198396 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.198694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198536 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" Nov 01 20:01:58.198694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198554 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" Nov 01 20:01:58.198694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.198563 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:01:58.202143 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.202117 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:01:58.202196 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.202156 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" Nov 01 20:01:58.245663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245638 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:01:58.245757 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245704 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.245802 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245752 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245905 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245921 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245912 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245939 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245952 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245975 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.246020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.245981 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-szdzx" Nov 01 20:01:58.246272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.246053 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:01:58.246272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.246071 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.246272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.246073 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.246272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.246072 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:01:58.247552 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.247537 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:01:58.247663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.247562 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" Nov 01 20:01:58.247663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.247563 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:01:58.248044 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.247988 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" Nov 01 20:01:58.248044 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.248009 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:01:58.248184 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.248088 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" Nov 01 20:01:58.248184 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.248135 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" Nov 01 20:01:58.288850 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-bin\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.288850 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-kubernetes\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.288982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-sys\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.288982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288886 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-bin\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.288982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnqc\" (UniqueName: \"kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.289087 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-sys\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.289087 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.288981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-kubernetes\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.289087 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-host-slash\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.289087 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-host-slash\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.289087 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcntf\" (UniqueName: \"kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.289087 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-systemd-units\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-k8s-cni-cncf-io\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-log-socket\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-slash\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289159 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-k8s-cni-cncf-io\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-run\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289207 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289221 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-systemd-units\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289255 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-run\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-hostroot\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-slash\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpb4w\" (UniqueName: \"kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289329 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-log-socket\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.289370 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289343 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-hostroot\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-kubelet\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-kubelet\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk569\" (UniqueName: \"kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289549 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-netns\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289588 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-var-lib-kubelet\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289612 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289639 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-netns\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-socket-dir-parent\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-system-cni-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289716 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-var-lib-kubelet\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.290205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289776 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-socket-dir-parent\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-system-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289830 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-modprobe-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289933 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-etc-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289989 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-etc-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289996 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.289989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-lib-modules\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290044 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-system-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-multus\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-lib-modules\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-kubelet\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-var-lib-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290131 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-modprobe-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291067 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290130 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-kubelet\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-systemd\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290166 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-var-lib-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-kubelet-config\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-cnibin\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290217 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-systemd\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290242 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-multus\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b696755-be38-4546-82e1-9b485bdef642-hosts-file\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290098 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-system-cni-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290266 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-cnibin\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqjw\" (UniqueName: \"kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290305 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-kubelet-config\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290215 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290344 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.291982 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-netd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290418 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b696755-be38-4546-82e1-9b485bdef642-hosts-file\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290432 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290466 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290478 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-netd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.290549 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.290618 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:01:58.790599793 +0000 UTC m=+2.262809033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290548 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cnibin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290632 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cnibin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysconfig\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-systemd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290777 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-bin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.292880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290845 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysconfig\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290896 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290925 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-dbus\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290966 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-conf-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290979 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-systemd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.290991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-multus-certs\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-bin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291125 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-dbus\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-multus-certs\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.293622 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291218 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-conf-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-node-log\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291786 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-host\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291828 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96z8\" (UniqueName: \"kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291852 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-node-log\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291880 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-host\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-netns\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-etc-kubernetes\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294159 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcr84\" (UniqueName: \"kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.291997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-conf\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292020 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-ovn\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j2h\" (UniqueName: \"kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sd5\" (UniqueName: \"kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-device-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-host\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292218 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-host\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-conf\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292339 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292370 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-ovn\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-netns\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-etc-kubernetes\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.292511 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.292556 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:58.792539171 +0000 UTC m=+2.264748411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:01:58.294815 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292822 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-device-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.295292 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.292925 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.300683 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.300520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.300683 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.300619 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.300683 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.300627 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.300854 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.300686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.300854 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.300709 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.300854 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.300826 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.353411 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.353363 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:01:58.353411 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.353390 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:01:58.353411 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.353404 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:01:58.353608 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.353497 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:58.853476014 +0000 UTC m=+2.325685251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:01:58.355643 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.355616 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sd5\" (UniqueName: \"kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.356147 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.356121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpb4w\" (UniqueName: \"kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.356911 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.356879 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96z8\" (UniqueName: \"kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.357109 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.357091 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcr84\" (UniqueName: \"kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.357820 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.357794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcntf\" (UniqueName: \"kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.358217 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.358198 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqjw\" (UniqueName: \"kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.358304 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.358250 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j2h\" (UniqueName: \"kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.358304 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.358296 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk569\" (UniqueName: \"kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.359364 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.359345 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnqc\" (UniqueName: \"kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.488381 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.488359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:01:58.497720 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.497699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:01:58.512661 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.512633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ldqdw" Nov 01 20:01:58.540348 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.540326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:01:58.547220 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.547205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:01:58.574707 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.574687 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:01:58.604832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.604778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:01:58.618483 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.618461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:01:58.624092 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.624076 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:01:58.680446 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:58.680373 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2f737d93958347cc920d4e5916d818.slice/crio-1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837 WatchSource:0}: Error finding container 1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837: Status 404 returned error can't find the container with id 1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837 Nov 01 20:01:58.685739 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.685722 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:01:58.691433 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:58.691411 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f167604710ff032b6c087eabbd79a27.slice/crio-23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5 WatchSource:0}: Error finding container 23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5: Status 404 returned error can't find the container with id 23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5 Nov 01 20:01:58.795827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.795806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:58.795910 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.795840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:58.795950 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.795930 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:01:58.795988 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.795948 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:01:58.795988 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.795973 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:59.795959551 +0000 UTC m=+3.268168777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:01:58.796061 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.796000 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:01:59.795982593 +0000 UTC m=+3.268191819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:01:58.877721 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.877671 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:58.896915 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.896891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:01:58.897005 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.896993 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:01:58.897048 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.897008 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:01:58.897048 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.897017 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:01:58.897115 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:58.897055 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:01:59.897045476 +0000 UTC m=+3.369254716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:01:58.965504 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:58.965479 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:59.004081 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.004041 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"6f167604710ff032b6c087eabbd79a27","Type":"ContainerStarted","Data":"23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5"} Nov 01 20:01:59.005168 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.005146 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"ec2f737d93958347cc920d4e5916d818","Type":"ContainerStarted","Data":"1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837"} Nov 01 20:01:59.249173 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.249089 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:58 +0000 UTC" deadline="2028-04-20 23:23:00.013131265 +0000 UTC" Nov 01 20:01:59.249173 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.249129 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12867h21m0.764006054s" Nov 01 20:01:59.357912 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.357887 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55e75ab_72d7_43bb_b1ff_e385180a3c70.slice/crio-fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7 WatchSource:0}: Error finding container fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7: Status 404 returned error can't find the container with id fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7 Nov 01 20:01:59.380332 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.380311 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b90e96_6395_4931_8526_2e9dcbc72e3f.slice/crio-32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e WatchSource:0}: Error finding container 32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e: Status 404 returned error can't find the container with id 32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e Nov 01 20:01:59.404674 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.404654 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cec48ca_4b08_41d1_bfad_2df7138f4b90.slice/crio-3548e20dbb396f7880de96d9462ee7e2cf8998755cd22f782da5b3b4e0733651 WatchSource:0}: Error finding container 3548e20dbb396f7880de96d9462ee7e2cf8998755cd22f782da5b3b4e0733651: Status 404 returned error can't find the container with id 3548e20dbb396f7880de96d9462ee7e2cf8998755cd22f782da5b3b4e0733651 Nov 01 20:01:59.412899 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.412882 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:01:59.476066 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.476045 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0effeb7a_2a88_48ad_9df6_4cd132b14052.slice/crio-908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665 WatchSource:0}: Error finding container 908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665: Status 404 returned error can't find the container with id 908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665 Nov 01 20:01:59.517165 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.517146 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b696755_be38_4546_82e1_9b485bdef642.slice/crio-6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390 WatchSource:0}: Error finding container 6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390: Status 404 returned error can't find the container with id 6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390 Nov 01 20:01:59.517946 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.517924 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902e2a0d_1e93_45e1_87cf_fd428a800f92.slice/crio-8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a WatchSource:0}: Error finding container 8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a: Status 404 returned error can't find the container with id 8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a Nov 01 20:01:59.571440 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.571420 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48faedf2_1476_4c1a_842d_e4a6e5ceeb82.slice/crio-27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de WatchSource:0}: Error finding container 27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de: Status 404 returned error can't find the container with id 27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de Nov 01 20:01:59.574364 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.574342 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f1d631_6c85_4c27_b21e_9ddc9c71db4e.slice/crio-775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2 WatchSource:0}: Error finding container 775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2: Status 404 returned error can't find the container with id 775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2 Nov 01 20:01:59.577583 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:01:59.577564 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d65d12_0f39_4bb4_9ced_cce27246077c.slice/crio-6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba WatchSource:0}: Error finding container 6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba: Status 404 returned error can't find the container with id 6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba Nov 01 20:01:59.805036 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.804965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:01:59.805174 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.805053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:01:59.805174 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.805141 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:01:59.805174 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.805170 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:01:59.805354 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.805218 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:01.805195553 +0000 UTC m=+5.277404794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:01:59.805354 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.805240 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:02:01.805229715 +0000 UTC m=+5.277438947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:01:59.906094 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.906053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:01:59.906581 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.906221 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:01:59.906581 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.906240 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:01:59.906581 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.906251 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:01:59.906581 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:01:59.906326 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:01.906311519 +0000 UTC m=+5.378520745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:00.000012 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:01:59.999986 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:00.000150 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.000016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:00.000150 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.000028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:00.000150 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:00.000135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:00.000309 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:00.000210 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:00.000309 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:00.000296 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:00.008440 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.008413 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4njz9" event={"ID":"0cec48ca-4b08-41d1-bfad-2df7138f4b90","Type":"ContainerStarted","Data":"3548e20dbb396f7880de96d9462ee7e2cf8998755cd22f782da5b3b4e0733651"} Nov 01 20:02:00.010050 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.010002 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ldqdw" event={"ID":"c55e75ab-72d7-43bb-b1ff-e385180a3c70","Type":"ContainerStarted","Data":"fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7"} Nov 01 20:02:00.012020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.011990 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zw4q4" event={"ID":"48faedf2-1476-4c1a-842d-e4a6e5ceeb82","Type":"ContainerStarted","Data":"27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de"} Nov 01 20:02:00.013482 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.013457 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" event={"ID":"902e2a0d-1e93-45e1-87cf-fd428a800f92","Type":"ContainerStarted","Data":"8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a"} Nov 01 20:02:00.015109 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.015061 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerStarted","Data":"908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665"} Nov 01 20:02:00.016954 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.016930 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e"} Nov 01 20:02:00.018484 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.018463 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5z7h" event={"ID":"b2d65d12-0f39-4bb4-9ced-cce27246077c","Type":"ContainerStarted","Data":"6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba"} Nov 01 20:02:00.019801 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.019775 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8wmlv" event={"ID":"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e","Type":"ContainerStarted","Data":"775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2"} Nov 01 20:02:00.021207 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.021182 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qc9m9" event={"ID":"8b696755-be38-4546-82e1-9b485bdef642","Type":"ContainerStarted","Data":"6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390"} Nov 01 20:02:00.249346 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.249253 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:58 +0000 UTC" deadline="2028-06-07 14:46:48.574035197 +0000 UTC" Nov 01 20:02:00.249346 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:00.249304 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14010h44m48.324734612s" Nov 01 20:02:01.825302 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:01.823469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:01.825302 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:01.823522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:01.825302 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.823628 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:01.825302 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.823685 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:05.823664818 +0000 UTC m=+9.295874069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:01.825302 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.823774 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:01.825302 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.823813 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:02:05.82379825 +0000 UTC m=+9.296007483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:01.924943 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:01.924638 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:01.925185 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.925167 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:01.925252 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.925224 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:01.925252 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.925242 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:01.925362 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.925301 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:05.925282439 +0000 UTC m=+9.397491679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:01.999915 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:01.999463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:01.999915 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.999589 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:01.999915 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:01.999648 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:01.999915 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.999742 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:01.999915 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:01.999793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:01.999915 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:01.999875 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:02.046405 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:02.046367 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"6f167604710ff032b6c087eabbd79a27","Type":"ContainerStarted","Data":"39c94a5e7dafbc89305d7e780a7654f3fa9a2878b6bb328d1a3f92112222fbcc"} Nov 01 20:02:02.050940 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:02.050365 2573 generic.go:356] "Generic (PLEG): container finished" podID="ec2f737d93958347cc920d4e5916d818" containerID="bc6296c1649bd5108fb4124fd3093edae08faad6af36d5c471879182568bb22e" exitCode=0 Nov 01 20:02:02.050940 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:02.050412 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"ec2f737d93958347cc920d4e5916d818","Type":"ContainerDied","Data":"bc6296c1649bd5108fb4124fd3093edae08faad6af36d5c471879182568bb22e"} Nov 01 20:02:02.290098 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:02.290025 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" podStartSLOduration=5.290006027 podStartE2EDuration="5.290006027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:02.168763601 +0000 UTC m=+5.640972852" watchObservedRunningTime="2026-11-01 20:02:02.290006027 +0000 UTC m=+5.762215279" Nov 01 20:02:03.069760 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:03.069699 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"ec2f737d93958347cc920d4e5916d818","Type":"ContainerStarted","Data":"853be52b889669b5e952965bababa6db527e6efd4593d8fe7883ed0285a773f9"} Nov 01 20:02:03.183145 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:03.182974 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" podStartSLOduration=6.182953078 podStartE2EDuration="6.182953078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:03.182175149 +0000 UTC m=+6.654384396" watchObservedRunningTime="2026-11-01 20:02:03.182953078 +0000 UTC m=+6.655162325" Nov 01 20:02:04.000033 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:03.999579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:04.000033 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:03.999725 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:04.000033 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:03.999801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:04.000033 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:03.999873 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:04.000033 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:03.999923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:04.000033 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:03.999987 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:04.073331 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:04.073296 2573 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="3c1afb9ed6c59fb51a39547376e7102edb6ff7ee861a2cc366bcc3207ac7c892" exitCode=0 Nov 01 20:02:04.074418 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:04.074393 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerDied","Data":"3c1afb9ed6c59fb51a39547376e7102edb6ff7ee861a2cc366bcc3207ac7c892"} Nov 01 20:02:05.086777 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:05.086703 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5z7h" event={"ID":"b2d65d12-0f39-4bb4-9ced-cce27246077c","Type":"ContainerStarted","Data":"499ae02ac9be3410b371f319e0635492f65dba0ec7ebd9a82edeb87fa35dfb23"} Nov 01 20:02:05.859272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:05.859198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:05.859426 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:05.859287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:05.859426 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.859405 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:05.859530 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.859461 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:02:13.859443857 +0000 UTC m=+17.331653098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:05.859530 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.859528 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:05.859657 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.859559 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:13.85954761 +0000 UTC m=+17.331756848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:05.960174 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:05.959854 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:05.960174 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.960035 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:05.960174 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.960057 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:05.960174 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.960069 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:05.960174 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.960126 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:13.96010738 +0000 UTC m=+17.432316623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:05.999684 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:05.999659 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:05.999809 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.999785 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:05.999886 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:05.999872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:05.999969 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:05.999951 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:06.000025 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:06.000018 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:06.000120 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:06.000100 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:06.094279 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:06.093661 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zw4q4" event={"ID":"48faedf2-1476-4c1a-842d-e4a6e5ceeb82","Type":"ContainerStarted","Data":"3543036de969b24aa4f73f185f20e3c8305c7781ea52a8e8741c586af7b613eb"} Nov 01 20:02:06.097456 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:06.097386 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" event={"ID":"902e2a0d-1e93-45e1-87cf-fd428a800f92","Type":"ContainerStarted","Data":"929c9c29d712e332771a60911f566c7a70bfc851f081f8741f43e420d5fc39ca"} Nov 01 20:02:06.206811 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:06.206014 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c5z7h" podStartSLOduration=4.066537118 podStartE2EDuration="9.205994335s" totalImagesPullingTime="5.139457217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:05.202858278 +0000 UTC m=+8.675067526" watchObservedRunningTime="2026-11-01 20:02:06.205994335 +0000 UTC m=+9.678203582" Nov 01 20:02:06.206811 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:06.206259 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zw4q4" podStartSLOduration=3.309334552 podStartE2EDuration="9.206251324s" totalImagesPullingTime="5.896916772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:06.20566163 +0000 UTC m=+9.677870896" watchObservedRunningTime="2026-11-01 20:02:06.206251324 +0000 UTC m=+9.678460573" Nov 01 20:02:07.517089 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:07.516564 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:02:07.518086 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:07.517871 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:02:07.999648 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:07.999606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:07.999648 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:07.999648 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:07.999864 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:07.999740 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:07.999864 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:07.999807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:07.999952 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:07.999898 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:08.000001 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:07.999972 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:08.302584 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:08.302417 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:02:08.942185 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:08.942019 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:02:08.302439642Z","UUID":"74ba8282-77bf-4b3d-8ede-59996cd9e21b","Handler":null,"Name":"","Endpoint":""} Nov 01 20:02:08.945323 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:08.945301 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:02:08.945452 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:08.945343 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:02:09.104950 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.104918 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qc9m9" event={"ID":"8b696755-be38-4546-82e1-9b485bdef642","Type":"ContainerStarted","Data":"5bd83b9e3d08c3bec7657376fd5a09abb75df82ce8b3687171ab4d4ad8204b5f"} Nov 01 20:02:09.106693 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.106666 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ldqdw" event={"ID":"c55e75ab-72d7-43bb-b1ff-e385180a3c70","Type":"ContainerStarted","Data":"6626c3cf266ea7ec9bb393dd5ccd91bb7131a2bc96b24c67189f9abbce4c47bd"} Nov 01 20:02:09.108920 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.108852 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" event={"ID":"902e2a0d-1e93-45e1-87cf-fd428a800f92","Type":"ContainerStarted","Data":"54a12e7b067f42c134208e707d099b8bc2074bf228c6be106e119a3146b4e438"} Nov 01 20:02:09.111264 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.111241 2573 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="c45affffaf27c13b174d2d0da7581ba4b76031c250f846d20795e25135652f56" exitCode=0 Nov 01 20:02:09.111397 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.111270 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerDied","Data":"c45affffaf27c13b174d2d0da7581ba4b76031c250f846d20795e25135652f56"} Nov 01 20:02:09.216629 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.216541 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qc9m9" podStartSLOduration=3.550059831 podStartE2EDuration="12.216525438s" totalImagesPullingTime="8.666465607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:09.216199692 +0000 UTC m=+12.688408940" watchObservedRunningTime="2026-11-01 20:02:09.216525438 +0000 UTC m=+12.688734686" Nov 01 20:02:09.330151 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:09.330101 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ldqdw" podStartSLOduration=3.579653213 podStartE2EDuration="12.330089734s" totalImagesPullingTime="8.750436521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:09.329693697 +0000 UTC m=+12.801902942" watchObservedRunningTime="2026-11-01 20:02:09.330089734 +0000 UTC m=+12.802298980" Nov 01 20:02:10.000217 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.000046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:10.000575 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.000067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:10.000575 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:10.000336 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:10.000575 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.000113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:10.000575 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:10.000425 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:10.000575 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:10.000548 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:10.114734 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.114701 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8wmlv" event={"ID":"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e","Type":"ContainerStarted","Data":"c751b445d89b856e9932b97bca9cd8496cbda33c74dcf108c7a11f6c106e899c"} Nov 01 20:02:10.116115 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.116088 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4njz9" event={"ID":"0cec48ca-4b08-41d1-bfad-2df7138f4b90","Type":"ContainerStarted","Data":"5dc49ef82094c6ae23bf67125ec97e6c38eab8928c8dcc2b6037357a83ed87f3"} Nov 01 20:02:10.117954 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.117933 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" event={"ID":"902e2a0d-1e93-45e1-87cf-fd428a800f92","Type":"ContainerStarted","Data":"83bbdf51c61e13242eccb07c5ed7c5fe565b05e834b5ed066b0b644644a6385e"} Nov 01 20:02:10.225822 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.225781 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8wmlv" podStartSLOduration=4.981864813 podStartE2EDuration="13.225768981s" totalImagesPullingTime="8.243904168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:10.225394044 +0000 UTC m=+13.697603289" watchObservedRunningTime="2026-11-01 20:02:10.225768981 +0000 UTC m=+13.697978220" Nov 01 20:02:10.339564 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.339493 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" podStartSLOduration=3.263267493 podStartE2EDuration="13.339482833s" totalImagesPullingTime="10.07621534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:10.339027053 +0000 UTC m=+13.811236298" watchObservedRunningTime="2026-11-01 20:02:10.339482833 +0000 UTC m=+13.811692078" Nov 01 20:02:10.451670 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:10.451612 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4njz9" podStartSLOduration=3.122611965 podStartE2EDuration="13.451600061s" totalImagesPullingTime="10.328988096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:10.451298239 +0000 UTC m=+13.923507485" watchObservedRunningTime="2026-11-01 20:02:10.451600061 +0000 UTC m=+13.923809307" Nov 01 20:02:11.124593 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:11.124559 2573 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="7af9afa9774b9bedd33ebca95bfc9c001e26c0a6d95dfca4838448ab61b98c1d" exitCode=0 Nov 01 20:02:11.125095 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:11.124663 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerDied","Data":"7af9afa9774b9bedd33ebca95bfc9c001e26c0a6d95dfca4838448ab61b98c1d"} Nov 01 20:02:11.999984 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:11.999922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:11.999984 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:11.999943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:12.000199 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:12.000022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:12.000199 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:12.000039 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:12.000199 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:12.000147 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:12.000316 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:12.000233 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:13.131987 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.131954 2573 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="e25c61da0b3b5f6f5f190fa15899ff3f1fd18c1c8f9646ad6c3e65c0e0249eae" exitCode=0 Nov 01 20:02:13.132651 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.131998 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerDied","Data":"e25c61da0b3b5f6f5f190fa15899ff3f1fd18c1c8f9646ad6c3e65c0e0249eae"} Nov 01 20:02:13.921963 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.921925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:13.922136 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.921980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:13.922136 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.922109 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:13.922219 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.922165 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.922152335 +0000 UTC m=+33.394361559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:13.922590 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.922572 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:13.922686 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.922625 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:02:29.922609555 +0000 UTC m=+33.394818779 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:13.999679 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.999649 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:13.999847 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.999683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:13.999847 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:13.999696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:13.999847 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.999769 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:13.999963 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.999893 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:14.000016 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:13.999997 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:14.022881 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:14.022850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:14.023006 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:14.022981 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:14.023006 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:14.022994 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:14.023006 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:14.023002 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:14.023122 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:14.023054 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:02:30.023039232 +0000 UTC m=+33.495248458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:15.999839 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:15.999730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:16.000147 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:15.999761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:16.000147 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:15.999911 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:16.000147 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:15.999978 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:16.000147 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:15.999787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:16.000147 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:16.000037 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:16.140427 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.140405 2573 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="6f082ebaab1f2cf4c76d7cd410f5d32f6022d55e3cda43d1dcd265655d62e533" exitCode=0 Nov 01 20:02:16.140561 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.140485 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerDied","Data":"6f082ebaab1f2cf4c76d7cd410f5d32f6022d55e3cda43d1dcd265655d62e533"} Nov 01 20:02:16.143163 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.143146 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"5d071af444406c363a78d589b1c7e4a42644c78854dc6812aad52f2da0aa7ff9"} Nov 01 20:02:16.143244 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.143197 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"3736f09d8896eda6d6f9c9b23621d61e4a7231cbbb5c8f05d04b1f5235a6e1fc"} Nov 01 20:02:16.143244 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.143228 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"a70b6912784372c069a8293f2e1769998d0ca70c88dbcd7c24711192de817ee2"} Nov 01 20:02:16.143355 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.143257 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"15443d4ebdc862bc2e9d677dceee94865729cdbe1242c87ae599ecd18c7257fa"} Nov 01 20:02:16.143355 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:16.143284 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"066f8f766e51ebd69c0214498f9d404c065f149fc2cf52c033c84f7ed107b75f"} Nov 01 20:02:17.148118 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:17.147887 2573 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="df41f5056a9e85c1be7e4409891609efd74e2f9ae54c663a4b93d8fcd69ba89a" exitCode=0 Nov 01 20:02:17.148530 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:17.147967 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerDied","Data":"df41f5056a9e85c1be7e4409891609efd74e2f9ae54c663a4b93d8fcd69ba89a"} Nov 01 20:02:17.151312 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:17.151288 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"e30f6e20b1a2955ccd78c9879d8a983212d666bb0d454bdb06ca9aea0d2883de"} Nov 01 20:02:18.000130 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:18.000106 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:18.000287 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:18.000138 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:18.000287 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:18.000139 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:18.000287 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:18.000213 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:18.000384 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:18.000290 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:18.000384 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:18.000365 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:18.157621 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:18.157593 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" event={"ID":"0effeb7a-2a88-48ad-9df6-4cd132b14052","Type":"ContainerStarted","Data":"f3c48edf4b5d058c5d5e1536c1621fe84396ae0dc1806ba9e4efdb2a2a732caa"} Nov 01 20:02:18.274729 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:18.274688 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7zvvr" podStartSLOduration=7.814965208 podStartE2EDuration="21.274676218s" totalImagesPullingTime="13.45971101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:18.274129121 +0000 UTC m=+21.746338384" watchObservedRunningTime="2026-11-01 20:02:18.274676218 +0000 UTC m=+21.746885463" Nov 01 20:02:19.163341 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:19.163307 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"584c02d065e9c1aafae16b858e0df5c9f1f9a24cc466d4dd827625f5717e9db2"} Nov 01 20:02:19.999760 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:19.999730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:19.999970 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:19.999801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:19.999970 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:19.999826 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:19.999970 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:19.999909 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:19.999970 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:19.999949 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:20.000096 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:20.000018 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:20.661290 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:20.661214 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:02:20.661979 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:20.661333 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:20.661979 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:20.661756 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:02:21.170153 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:21.170118 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" event={"ID":"29b90e96-6395-4931-8526-2e9dcbc72e3f","Type":"ContainerStarted","Data":"00b9394db9f4807daa8c5fbdf56864ce97604576a822aff379b4d02c4bd83b20"} Nov 01 20:02:21.170442 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:21.170418 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:02:21.170511 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:21.170483 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:02:21.185558 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:21.185536 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:02:21.185684 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:21.185626 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:02:21.475251 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:21.475164 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" podStartSLOduration=8.01249373 podStartE2EDuration="24.475152352s" totalImagesPullingTime="16.462658622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:21.425289283 +0000 UTC m=+24.897498530" watchObservedRunningTime="2026-11-01 20:02:21.475152352 +0000 UTC m=+24.947361649" Nov 01 20:02:22.000213 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:22.000180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:22.000213 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:22.000209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:22.000702 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:22.000295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:22.000702 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:22.000303 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:22.000702 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:22.000362 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:22.000702 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:22.000443 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:22.172049 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:22.172020 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:23.026597 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.026397 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tn5w8"] Nov 01 20:02:23.027039 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.026685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:23.027039 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:23.026774 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:23.029314 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.029288 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5w6p"] Nov 01 20:02:23.029428 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.029379 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:23.029501 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:23.029480 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:23.076181 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.076157 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-88xp6"] Nov 01 20:02:23.076278 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.076233 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:23.076327 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:23.076314 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:23.173667 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:23.173631 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:25.000135 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.000107 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:25.000135 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.000131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:25.000573 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.000146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:25.000573 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:25.000228 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:25.000573 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:25.000306 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:25.000573 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:25.000362 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:25.093793 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.093771 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:02:25.093949 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.093937 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:02:25.104080 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.104043 2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" podUID="29b90e96-6395-4931-8526-2e9dcbc72e3f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 01 20:02:25.111923 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:25.111900 2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" podUID="29b90e96-6395-4931-8526-2e9dcbc72e3f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 01 20:02:27.000309 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:27.000284 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:27.000721 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:27.000368 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:27.000721 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:27.000452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:27.000721 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:27.000536 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:27.000721 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:27.000570 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:27.000721 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:27.000622 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:28.999890 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:28.999862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:29.000281 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:28.999900 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:29.000281 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:28.999954 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:29.000281 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:28.999991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:29.000281 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:29.000064 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:29.000281 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:29.000153 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:29.659271 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:29.659244 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qc9m9_8b696755-be38-4546-82e1-9b485bdef642/dns-node-resolver/0.log" Nov 01 20:02:29.939542 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:29.939467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:29.939542 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:29.939505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:29.939740 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:29.939602 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:29.939740 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:29.939664 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:29.939740 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:29.939670 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret podName:ffffa9d9-1fea-40cb-85c6-1a1edb49470d nodeName:}" failed. No retries permitted until 2026-11-01 20:03:01.939653318 +0000 UTC m=+65.411862544 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret") pod "global-pull-secret-syncer-tn5w8" (UID: "ffffa9d9-1fea-40cb-85c6-1a1edb49470d") : object "kube-system"/"original-pull-secret" not registered Nov 01 20:02:29.939740 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:29.939711 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs podName:afc73791-d8d2-4a7e-84c2-eb514db3fee1 nodeName:}" failed. No retries permitted until 2026-11-01 20:03:01.939699415 +0000 UTC m=+65.411908639 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs") pod "network-metrics-daemon-88xp6" (UID: "afc73791-d8d2-4a7e-84c2-eb514db3fee1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 20:02:30.040426 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:30.040400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:30.040794 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.040516 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 20:02:30.040794 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.040530 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 20:02:30.040794 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.040538 2573 projected.go:196] Error preparing data for projected volume kube-api-access-kjhhp for pod openshift-network-diagnostics/network-check-target-h5w6p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:30.040794 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.040576 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp podName:4a46610c-94b6-4ec2-a351-e83bf7b95032 nodeName:}" failed. No retries permitted until 2026-11-01 20:03:02.040564339 +0000 UTC m=+65.512773575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kjhhp" (UniqueName: "kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp") pod "network-check-target-h5w6p" (UID: "4a46610c-94b6-4ec2-a351-e83bf7b95032") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 20:02:30.494029 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:30.494002 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zw4q4_48faedf2-1476-4c1a-842d-e4a6e5ceeb82/node-ca/0.log" Nov 01 20:02:30.999187 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:30.999160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:30.999338 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:30.999160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:30.999338 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.999256 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:30.999338 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:30.999268 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:30.999338 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.999328 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:30.999468 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:30.999392 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:33.000305 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:33.000218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:33.000305 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:33.000275 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:33.000305 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:33.000294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:33.000871 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:33.000414 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:33.000871 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:33.000456 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:33.000871 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:33.000499 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:35.000175 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:35.000146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:35.000175 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:35.000164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:35.000673 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:35.000146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:35.000673 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:35.000235 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:35.000673 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:35.000321 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:35.000673 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:35.000409 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:37.000780 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:37.000752 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:37.001134 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:37.000825 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:37.001134 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:37.000843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:37.001134 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:37.000938 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:37.001134 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:37.000988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:37.001134 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:37.001042 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:38.999915 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:38.999886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:39.000277 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:38.999978 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:39.000277 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:38.999886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:39.000277 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:39.000049 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:39.000277 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:38.999886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:39.000277 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:39.000123 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:40.999968 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:40.999943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:41.000386 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:40.999974 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:41.000386 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:41.000041 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:41.000386 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:41.000123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:41.000386 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:41.000118 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:41.000386 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:41.000211 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:42.999584 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:42.999555 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:42.999584 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:42.999565 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:43.000009 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:42.999689 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:43.000009 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:42.999731 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:43.000009 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:42.999756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:43.000009 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:42.999822 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:44.999880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:44.999848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:44.999880 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:44.999870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:45.000278 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:44.999885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:45.000278 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:44.999967 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-88xp6" podUID="afc73791-d8d2-4a7e-84c2-eb514db3fee1" Nov 01 20:02:45.000278 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:45.000085 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tn5w8" podUID="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" Nov 01 20:02:45.000278 ip-10-0-138-177 kubenswrapper[2573]: E1101 20:02:45.000154 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h5w6p" podUID="4a46610c-94b6-4ec2-a351-e83bf7b95032" Nov 01 20:02:46.339339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.339316 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeReady" Nov 01 20:02:46.339779 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.339402 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 20:02:46.494088 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.494065 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rvk4h"] Nov 01 20:02:46.495694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.495677 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nbcrt"] Nov 01 20:02:46.495799 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.495784 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.497275 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.497256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.504496 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.503716 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hz4mt"] Nov 01 20:02:46.505391 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.505371 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvk4h"] Nov 01 20:02:46.505475 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.505415 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nbcrt"] Nov 01 20:02:46.505527 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.505492 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.549334 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.549313 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:02:46.549415 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.549394 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" Nov 01 20:02:46.549504 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.549487 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:02:46.549542 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.549494 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:02:46.549596 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.549584 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" Nov 01 20:02:46.549703 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.549690 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:02:46.554357 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.554343 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:02:46.557569 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.557556 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" Nov 01 20:02:46.559176 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.559155 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hz4mt"] Nov 01 20:02:46.652881 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.652785 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:02:46.652987 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.652895 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:02:46.652987 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.652905 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:02:46.652987 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.652972 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:02:46.661161 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.661228 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-crio-socket\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.661228 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.661228 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.661348 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjh4\" (UniqueName: \"kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.661348 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.661408 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fj4\" (UniqueName: \"kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.661408 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.661464 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.661545 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.661577 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.661556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w466g\" (UniqueName: \"kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.762669 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.762760 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.762760 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.762760 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762717 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w466g\" (UniqueName: \"kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-crio-socket\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762836 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjh4\" (UniqueName: \"kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.762994 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.762933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fj4\" (UniqueName: \"kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.763311 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.763005 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.763311 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.763158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-crio-socket\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.763311 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.763278 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.763450 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.763407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.763694 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.763677 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.766400 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.766377 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.766484 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.766386 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.766484 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.766407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.819744 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.819718 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fj4\" (UniqueName: \"kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:46.819836 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.819814 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w466g\" (UniqueName: \"kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:46.819972 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.819954 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjh4\" (UniqueName: \"kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:46.820793 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:46.820781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:02:47.002614 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.002538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:02:47.002772 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.002661 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:02:47.002849 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.002831 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:02:47.086687 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086663 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:02:47.086827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086711 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:02:47.086827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086744 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:02:47.086827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086772 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:02:47.086827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086788 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:02:47.086827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086816 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:02:47.087019 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.086834 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:02:47.110016 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.110000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:47.114265 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.114249 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:02:47.249489 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.249463 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hz4mt"] Nov 01 20:02:47.478434 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.478392 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rvk4h"] Nov 01 20:02:47.479580 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:47.479552 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nbcrt"] Nov 01 20:02:47.480991 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:02:47.480970 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007fa95_0178_4da4_a163_387a0528b3d1.slice/crio-ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8 WatchSource:0}: Error finding container ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8: Status 404 returned error can't find the container with id ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8 Nov 01 20:02:47.483471 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:02:47.483451 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6e1cad_750a_416c_9a32_81bf6a048e75.slice/crio-b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e WatchSource:0}: Error finding container b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e: Status 404 returned error can't find the container with id b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e Nov 01 20:02:48.221560 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:48.221522 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvk4h" event={"ID":"b007fa95-0178-4da4-a163-387a0528b3d1","Type":"ContainerStarted","Data":"ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8"} Nov 01 20:02:48.222963 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:48.222938 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4mt" event={"ID":"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d","Type":"ContainerStarted","Data":"0b499677278eeb7eec972a40fff58a7cb454ee0bd4e90f485395ae064a2084e5"} Nov 01 20:02:48.223089 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:48.222988 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4mt" event={"ID":"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d","Type":"ContainerStarted","Data":"bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36"} Nov 01 20:02:48.224022 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:48.223998 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nbcrt" event={"ID":"fd6e1cad-750a-416c-9a32-81bf6a048e75","Type":"ContainerStarted","Data":"b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e"} Nov 01 20:02:49.228990 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:49.228904 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4mt" event={"ID":"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d","Type":"ContainerStarted","Data":"eb13111b9d6caa659d87733db7db5bed84200e1757556b7321c9ee4df3153e27"} Nov 01 20:02:50.233885 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:50.233808 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvk4h" event={"ID":"b007fa95-0178-4da4-a163-387a0528b3d1","Type":"ContainerStarted","Data":"3671ad1f37219acf428cfe695c0bf8bfc8da31d32f51740b553d5727e8fb1746"} Nov 01 20:02:50.233885 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:50.233874 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rvk4h" event={"ID":"b007fa95-0178-4da4-a163-387a0528b3d1","Type":"ContainerStarted","Data":"9cf80a73a62d9de8f3ebd7418323d2ba47ec0a5d3112fe64d4a9298548ca0ce7"} Nov 01 20:02:50.234354 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:50.234118 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rvk4h" Nov 01 20:02:50.349218 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:50.349162 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rvk4h" podStartSLOduration=1.987658627 podStartE2EDuration="4.349148728s" totalImagesPullingTime="2.361490101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:50.348333704 +0000 UTC m=+53.820542952" watchObservedRunningTime="2026-11-01 20:02:50.349148728 +0000 UTC m=+53.821358331" Nov 01 20:02:51.237745 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:51.237715 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4mt" event={"ID":"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d","Type":"ContainerStarted","Data":"b016418a4a9fb283f0c0b465bb4cd74116f4a362a8d9a9885e0fc5c1e28395dd"} Nov 01 20:02:51.239237 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:51.239211 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nbcrt" event={"ID":"fd6e1cad-750a-416c-9a32-81bf6a048e75","Type":"ContainerStarted","Data":"93d34174c7f1088f263b27fe32d42e06b621a59b6c27ef2fb2275c82d2bbc936"} Nov 01 20:02:51.407339 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:51.407238 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hz4mt" podStartSLOduration=1.671158511 podStartE2EDuration="5.407222713s" totalImagesPullingTime="3.736064202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:51.40708313 +0000 UTC m=+54.879292376" watchObservedRunningTime="2026-11-01 20:02:51.407222713 +0000 UTC m=+54.879431961" Nov 01 20:02:51.524866 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:51.524741 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nbcrt" podStartSLOduration=2.299737336 podStartE2EDuration="5.524729686s" totalImagesPullingTime="3.22499235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:51.523845492 +0000 UTC m=+54.996054738" watchObservedRunningTime="2026-11-01 20:02:51.524729686 +0000 UTC m=+54.996938929" Nov 01 20:02:53.129324 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.129294 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dcf7fbf49-rdv7w"] Nov 01 20:02:53.131147 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.131132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.184043 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.184014 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 20:02:53.234200 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234168 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:02:53.234397 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234373 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-877r8\"" Nov 01 20:02:53.234515 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234418 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 20:02:53.234515 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234413 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 20:02:53.234515 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234447 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 20:02:53.234515 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234374 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:02:53.234515 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.234425 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 20:02:53.235213 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.235195 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dcf7fbf49-rdv7w"] Nov 01 20:02:53.312922 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.312890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.312922 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.312917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.313113 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.313015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.313113 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.313055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.313113 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.313073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.313260 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.313206 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.313260 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.313239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414489 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414489 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414740 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414740 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414740 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414740 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414740 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.414989 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.414819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.415272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.415246 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.415345 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.415271 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.415345 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.415334 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.416940 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.416922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.416992 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.416934 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.470805 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.470785 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:53.740816 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:53.740727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:02:54.107039 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:54.107010 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dcf7fbf49-rdv7w"] Nov 01 20:02:54.110170 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:02:54.110146 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6fa94a_74a8_46e6_a813_997f4f507da9.slice/crio-afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47 WatchSource:0}: Error finding container afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47: Status 404 returned error can't find the container with id afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47 Nov 01 20:02:54.247280 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:54.247251 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcf7fbf49-rdv7w" event={"ID":"0b6fa94a-74a8-46e6-a813-997f4f507da9","Type":"ContainerStarted","Data":"afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47"} Nov 01 20:02:55.113693 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:55.113661 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:02:56.837272 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.836250 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j"] Nov 01 20:02:56.838783 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.838761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:56.891091 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.891061 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:02:56.940953 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.940920 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:02:56.940953 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.940956 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 20:02:56.941157 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.941040 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:02:56.941157 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.941104 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2j5jb\"" Nov 01 20:02:56.941236 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.941189 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:02:56.942086 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.942057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:56.942225 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.942103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:56.942225 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.942176 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j"] Nov 01 20:02:56.942356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.942178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wh5x\" (UniqueName: \"kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:56.942356 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.942267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:56.994255 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.994228 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ll458"] Nov 01 20:02:56.999089 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:56.999068 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.042630 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.042814 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-sys\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.042814 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.042932 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-node-exporter-wtmp\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.042932 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.042932 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.043098 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.042961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.043098 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.043013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.044749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.043780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rgc\" (UniqueName: \"kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.044749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.043822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wh5x\" (UniqueName: \"kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.044749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.043849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-root\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.044749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.043877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.044749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.043907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.044749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.044712 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.045633 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.045608 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.045846 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.045825 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.067073 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.067050 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:02:57.067261 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.067239 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" Nov 01 20:02:57.067393 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.067373 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:02:57.102237 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.102182 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wh5x\" (UniqueName: \"kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.132623 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.132589 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:02:57.145048 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-sys\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145142 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145142 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-node-exporter-wtmp\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145142 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145285 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145144 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-sys\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145321 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-node-exporter-wtmp\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145361 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145425 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rgc\" (UniqueName: \"kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145474 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145442 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-root\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145474 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145568 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145508 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145653 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145626 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145715 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-root\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.145801 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.145783 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.146251 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.146223 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.147887 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.147866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.147998 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.147975 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.152950 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.152929 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:02:57.206322 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.206291 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rgc\" (UniqueName: \"kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.313043 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.313017 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:02:57.326039 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:02:57.326019 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf45e616_0778_4869_9066_7acad174e56a.slice/crio-3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c WatchSource:0}: Error finding container 3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c: Status 404 returned error can't find the container with id 3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c Nov 01 20:02:57.574697 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:57.574666 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j"] Nov 01 20:02:57.578267 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:02:57.578240 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb7e9a5_c890_47bf_994a_db739d7e2817.slice/crio-f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c WatchSource:0}: Error finding container f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c: Status 404 returned error can't find the container with id f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c Nov 01 20:02:58.260764 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:58.260722 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcf7fbf49-rdv7w" event={"ID":"0b6fa94a-74a8-46e6-a813-997f4f507da9","Type":"ContainerStarted","Data":"347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d"} Nov 01 20:02:58.261954 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:58.261930 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ll458" event={"ID":"af45e616-0778-4869-9066-7acad174e56a","Type":"ContainerStarted","Data":"3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c"} Nov 01 20:02:58.263552 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:58.263532 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" event={"ID":"3eb7e9a5-c890-47bf-994a-db739d7e2817","Type":"ContainerStarted","Data":"5d9c900c3298992ab0cb44649f57550b5fad31a189284bff8a1c545a2dae03ea"} Nov 01 20:02:58.263624 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:58.263572 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" event={"ID":"3eb7e9a5-c890-47bf-994a-db739d7e2817","Type":"ContainerStarted","Data":"84af42a8a754f1f78db1d4e86a0d10372b732e9c6fc6bc9acae65a345104463b"} Nov 01 20:02:58.263624 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:58.263591 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" event={"ID":"3eb7e9a5-c890-47bf-994a-db739d7e2817","Type":"ContainerStarted","Data":"f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c"} Nov 01 20:02:58.375323 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:58.374544 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7dcf7fbf49-rdv7w" podStartSLOduration=2.158224862 podStartE2EDuration="5.374528439s" totalImagesPullingTime="3.216303577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:58.374199721 +0000 UTC m=+61.846408960" watchObservedRunningTime="2026-11-01 20:02:58.374528439 +0000 UTC m=+61.846737762" Nov 01 20:02:59.273085 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.273052 2573 generic.go:356] "Generic (PLEG): container finished" podID="af45e616-0778-4869-9066-7acad174e56a" containerID="9fbfb6015e1e27fc25f2aadbbf492117a9dc927ccd7d39e74e9d4ed288673e5e" exitCode=0 Nov 01 20:02:59.273490 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.273126 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ll458" event={"ID":"af45e616-0778-4869-9066-7acad174e56a","Type":"ContainerDied","Data":"9fbfb6015e1e27fc25f2aadbbf492117a9dc927ccd7d39e74e9d4ed288673e5e"} Nov 01 20:02:59.721372 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.721342 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5867784ff5-9s6fg"] Nov 01 20:02:59.724723 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.724707 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766627 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766725 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766769 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766803 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766777 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766832 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766890 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766923 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttkdq\" (UniqueName: \"kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.766952 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.766928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.767032 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.767009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.777298 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777280 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 20:02:59.777493 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777467 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 20:02:59.777493 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777471 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 20:02:59.777493 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777500 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bspnd5hul0il6\"" Nov 01 20:02:59.777678 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777474 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 20:02:59.777678 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777530 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 20:02:59.777678 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.777573 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rxxgh\"" Nov 01 20:02:59.787998 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.787972 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5867784ff5-9s6fg"] Nov 01 20:02:59.867750 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.867750 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.867895 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttkdq\" (UniqueName: \"kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.867895 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.867895 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.868025 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.868025 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.868025 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.867969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.868025 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.868013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.868431 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.868402 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.868577 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.868554 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.870504 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.870480 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.870596 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.870487 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.870820 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.870797 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.870887 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.870815 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.870942 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.870879 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.871204 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.871188 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:02:59.923887 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:02:59.923859 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttkdq\" (UniqueName: \"kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:00.046037 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.046004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:00.241331 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.241305 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:00.278151 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.278118 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ll458" event={"ID":"af45e616-0778-4869-9066-7acad174e56a","Type":"ContainerStarted","Data":"b79bfa7a37e1dfeed5335122672a2a1be657fd5c7775d31ad1e10f67a272e2d9"} Nov 01 20:03:00.278540 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.278172 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ll458" event={"ID":"af45e616-0778-4869-9066-7acad174e56a","Type":"ContainerStarted","Data":"7c607bdc7bb2b0a9e7927f9a21856040325d54699347e177e73619f3d7d9f3ff"} Nov 01 20:03:00.280096 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.280074 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" event={"ID":"3eb7e9a5-c890-47bf-994a-db739d7e2817","Type":"ContainerStarted","Data":"d2fbf8181127299cb9fd5c410a9f7fa5383caf2aba02d9591339524596f04d9a"} Nov 01 20:03:00.470353 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.470236 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" podStartSLOduration=2.895342624 podStartE2EDuration="4.470218664s" totalImagesPullingTime="1.57487604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:00.469181951 +0000 UTC m=+63.941391207" watchObservedRunningTime="2026-11-01 20:03:00.470218664 +0000 UTC m=+63.942427912" Nov 01 20:03:00.517422 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.517390 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5867784ff5-9s6fg"] Nov 01 20:03:00.524812 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:03:00.524787 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d12c00_e8a7_4ac4_af79_b618b391618e.slice/crio-9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb WatchSource:0}: Error finding container 9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb: Status 404 returned error can't find the container with id 9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb Nov 01 20:03:00.586708 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:00.586657 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ll458" podStartSLOduration=3.047222483 podStartE2EDuration="4.586628557s" totalImagesPullingTime="1.464828288s" totalInitContainerRuntime="74.577786ms" isStatefulPod=true podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:00.585303791 +0000 UTC m=+64.057513037" watchObservedRunningTime="2026-11-01 20:03:00.586628557 +0000 UTC m=+64.058837803" Nov 01 20:03:01.285233 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.285196 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb"} Nov 01 20:03:01.759592 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.759560 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-z6p67"] Nov 01 20:03:01.762645 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.762622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.783825 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.783804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcg8\" (UniqueName: \"kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.783934 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.783856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.819777 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.819748 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-z6p67"] Nov 01 20:03:01.871503 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.871481 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-99s87\"" Nov 01 20:03:01.871503 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.871482 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 20:03:01.884377 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.884355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcg8\" (UniqueName: \"kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.884467 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.884409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.886766 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.886744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.940299 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.940276 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcg8\" (UniqueName: \"kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:01.985090 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.985066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:01.985198 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:01.985099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:02.036779 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.036717 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:03:02.036870 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.036775 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:03:02.047909 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.047882 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:02.048006 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.047922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:02.074027 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.074004 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:02.086352 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.086313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:02.137729 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.137706 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:03:02.198264 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.198234 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:03:02.209940 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.209908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:02.364517 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.364456 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:03:02.370274 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.370259 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:03:02.373298 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.373281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:02.375459 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.375441 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:03:02.380031 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.380014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:02.384860 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.384846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:02.441119 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.440530 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-z6p67"] Nov 01 20:03:02.757754 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.757712 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h5w6p"] Nov 01 20:03:02.761154 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:03:02.761128 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a46610c_94b6_4ec2_a351_e83bf7b95032.slice/crio-993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7 WatchSource:0}: Error finding container 993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7: Status 404 returned error can't find the container with id 993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7 Nov 01 20:03:02.808020 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.807993 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tn5w8"] Nov 01 20:03:02.810827 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:02.810801 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-88xp6"] Nov 01 20:03:02.811667 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:03:02.811629 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffffa9d9_1fea_40cb_85c6_1a1edb49470d.slice/crio-12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de WatchSource:0}: Error finding container 12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de: Status 404 returned error can't find the container with id 12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de Nov 01 20:03:02.815227 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:03:02.815205 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc73791_d8d2_4a7e_84c2_eb514db3fee1.slice/crio-31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144 WatchSource:0}: Error finding container 31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144: Status 404 returned error can't find the container with id 31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144 Nov 01 20:03:03.293481 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.293437 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tn5w8" event={"ID":"ffffa9d9-1fea-40cb-85c6-1a1edb49470d","Type":"ContainerStarted","Data":"12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de"} Nov 01 20:03:03.294864 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.294835 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" event={"ID":"78496de5-ccaa-448e-8683-52ea3c9e5de1","Type":"ContainerStarted","Data":"fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae"} Nov 01 20:03:03.296120 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.296090 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88xp6" event={"ID":"afc73791-d8d2-4a7e-84c2-eb514db3fee1","Type":"ContainerStarted","Data":"31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144"} Nov 01 20:03:03.297352 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.297330 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5w6p" event={"ID":"4a46610c-94b6-4ec2-a351-e83bf7b95032","Type":"ContainerStarted","Data":"993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7"} Nov 01 20:03:03.742305 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.742018 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:03.742305 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.742079 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:03.747193 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:03.747167 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:04.305709 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.305670 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"00cb83834001e0f259d5f1031693f92339ac1e41b0a05474a11f7eff937cc53f"} Nov 01 20:03:04.305868 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.305737 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"8050bec5de84f4a33ec7a4122b9439fe328cdf0407676b3f78513c50ea34a692"} Nov 01 20:03:04.305868 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.305766 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"b3b0ee20318973fb3c26c40d8bfd144a71908ec9792f026b87bf69a7df857e8f"} Nov 01 20:03:04.311669 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.310698 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:04.687732 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.687523 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c49869bf6-pj89j"] Nov 01 20:03:04.694501 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.694475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.751581 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.751482 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c49869bf6-pj89j"] Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806369 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.806663 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.806614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.851294 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.849479 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 20:03:04.908095 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908205 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908403 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908403 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.908403 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.908263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.909357 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.909334 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.911369 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.911238 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.911369 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.911324 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.911679 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.911617 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.911751 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.911686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.912061 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.912039 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.912150 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.912108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:04.966193 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:04.966168 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:05.018841 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.018667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:05.310328 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.310288 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" event={"ID":"78496de5-ccaa-448e-8683-52ea3c9e5de1","Type":"ContainerStarted","Data":"9c8c8a19b903894720da576ba2960bdcf761f1f9c98c4e7a6a2651d972936b90"} Nov 01 20:03:05.311304 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.311280 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:05.314031 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.314002 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88xp6" event={"ID":"afc73791-d8d2-4a7e-84c2-eb514db3fee1","Type":"ContainerStarted","Data":"681e5e2a719fbdad6dba649477e958e373d23b35a9fa23d09c849897686bd0f5"} Nov 01 20:03:05.314140 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.314074 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-88xp6" event={"ID":"afc73791-d8d2-4a7e-84c2-eb514db3fee1","Type":"ContainerStarted","Data":"0681afad353cab8dcc71d792d243955bf74b303a97f552e9ef8c8c1c61ff6c93"} Nov 01 20:03:05.317063 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.317037 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:05.416707 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.416677 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c49869bf6-pj89j"] Nov 01 20:03:05.419828 ip-10-0-138-177 kubenswrapper[2573]: W1101 20:03:05.419802 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a53dce_f4f6_42cc_8fe6_52b35c8bb09e.slice/crio-1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923 WatchSource:0}: Error finding container 1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923: Status 404 returned error can't find the container with id 1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923 Nov 01 20:03:05.425052 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.424884 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" podStartSLOduration=2.098074226 podStartE2EDuration="4.424866434s" totalImagesPullingTime="2.326792208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:05.42453878 +0000 UTC m=+68.896748027" watchObservedRunningTime="2026-11-01 20:03:05.424866434 +0000 UTC m=+68.897075681" Nov 01 20:03:05.538420 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:05.538369 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-88xp6" podStartSLOduration=66.459764793 podStartE2EDuration="1m8.538353193s" totalImagesPullingTime="2.0785884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:05.537022903 +0000 UTC m=+69.009232150" watchObservedRunningTime="2026-11-01 20:03:05.538353193 +0000 UTC m=+69.010562467" Nov 01 20:03:06.321323 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.321282 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49869bf6-pj89j" event={"ID":"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e","Type":"ContainerStarted","Data":"c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb"} Nov 01 20:03:06.321796 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.321342 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49869bf6-pj89j" event={"ID":"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e","Type":"ContainerStarted","Data":"1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923"} Nov 01 20:03:06.325312 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.325282 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"31478c3be358bf1294123d37c4a7aa701cf0599d18d05b267136176c4bf30f82"} Nov 01 20:03:06.325449 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.325357 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"18cf77eaa9106e84b2c6ff8740ceabe4584456e7b97a393849690a3912b6db64"} Nov 01 20:03:06.325449 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.325402 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" event={"ID":"78d12c00-e8a7-4ac4-af79-b618b391618e","Type":"ContainerStarted","Data":"e0bc0c1dd689f152eac309534d43d352e9b60d0d63a3cacbd5e488fe91edd280"} Nov 01 20:03:06.437161 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.437101 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c49869bf6-pj89j" podStartSLOduration=2.437084117 podStartE2EDuration="2.437084117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:06.4359767 +0000 UTC m=+69.908185946" watchObservedRunningTime="2026-11-01 20:03:06.437084117 +0000 UTC m=+69.909293365" Nov 01 20:03:06.554594 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:06.554530 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" podStartSLOduration=2.552604582 podStartE2EDuration="7.5545197s" totalImagesPullingTime="5.001915118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:06.553482651 +0000 UTC m=+70.025691899" watchObservedRunningTime="2026-11-01 20:03:06.5545197 +0000 UTC m=+70.026728941" Nov 01 20:03:07.330749 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:07.330716 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h5w6p" event={"ID":"4a46610c-94b6-4ec2-a351-e83bf7b95032","Type":"ContainerStarted","Data":"5ca7bc51f120e6f0fae33abd222097a3d08b305d4c50116d33e98decd05cf27e"} Nov 01 20:03:07.331216 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:07.331151 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:07.331692 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:07.331668 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:07.446646 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:07.446587 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h5w6p" podStartSLOduration=66.39863013 podStartE2EDuration="1m10.446569843s" totalImagesPullingTime="4.047939713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:07.44626733 +0000 UTC m=+70.918476576" watchObservedRunningTime="2026-11-01 20:03:07.446569843 +0000 UTC m=+70.918779091" Nov 01 20:03:08.342512 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:08.342483 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:08.688883 ip-10-0-138-177 kubenswrapper[2573]: I1101 20:03:08.688777 2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:08.688946 ip-10-0-138-177 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 20:03:08.704044 ip-10-0-138-177 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 20:03:08.704205 ip-10-0-138-177 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 20:03:08.704256 ip-10-0-138-177 systemd[1]: kubelet.service: Consumed 3.688s CPU time. Nov 01 20:03:08.719399 ip-10-0-138-177 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:03:08.779950 ip-10-0-138-177 kubenswrapper[5393]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:08.779950 ip-10-0-138-177 kubenswrapper[5393]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:03:08.779950 ip-10-0-138-177 kubenswrapper[5393]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:08.779950 ip-10-0-138-177 kubenswrapper[5393]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:08.782433 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782356 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.782433 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782431 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.782433 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782435 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.782433 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782438 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782442 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782445 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782448 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782451 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782454 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782456 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782459 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782462 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782465 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782467 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782471 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782473 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782476 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782479 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782481 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782484 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782486 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782489 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782492 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.782591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782495 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782497 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782500 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782508 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782511 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782515 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782519 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782523 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782526 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782528 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782531 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782534 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782540 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782543 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782545 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782548 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782551 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782554 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782557 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.783143 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782559 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782562 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782564 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782567 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782570 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782572 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782575 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782577 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782580 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782584 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782586 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782589 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782591 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782594 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782597 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782604 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782607 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782610 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782612 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782621 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.783594 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782624 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782626 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782629 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782632 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782648 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782654 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782658 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782663 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782667 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782671 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782673 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782676 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782679 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782682 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782684 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782688 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782691 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782693 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782696 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.784092 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782698 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782701 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782704 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782707 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782709 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782712 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782716 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782718 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782721 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782731 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782734 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782737 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782739 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782742 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782745 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782747 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782750 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782753 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782756 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.784556 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782759 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782761 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782764 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782767 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782769 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.782772 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783544 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783551 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783554 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783557 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783560 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783562 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783565 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783568 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783571 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783574 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783577 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783579 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783582 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783585 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.785045 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783588 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783591 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783593 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783597 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783600 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783602 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783605 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783608 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783610 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783613 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783616 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783621 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783623 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783626 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783628 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783631 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783650 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783654 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783656 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.785518 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783659 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783662 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783664 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783667 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783670 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783673 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783675 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783678 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783681 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783683 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783686 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783688 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783691 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783694 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783697 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783699 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783702 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783705 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783708 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783710 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.785978 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783713 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783716 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783720 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783724 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783729 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783732 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783735 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783738 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783742 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783746 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783750 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783754 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783758 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783762 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783766 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783770 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783775 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783779 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783783 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.786454 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783787 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783791 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783794 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783799 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783803 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783807 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783811 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783815 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783821 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783825 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783830 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783844 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783848 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783850 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783854 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783857 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783859 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783864 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783866 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783869 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.786959 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783871 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783874 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783877 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783880 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783883 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783887 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783891 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783895 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783897 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783900 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783903 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783906 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783909 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.783911 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784472 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784478 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784481 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784484 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784486 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784489 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.787463 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784492 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784495 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784497 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784506 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784509 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784511 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784514 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784517 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784521 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784527 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784530 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784534 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784537 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784539 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784542 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784545 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784547 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784550 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784552 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.787948 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784555 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784557 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784560 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784563 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784565 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784568 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784571 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784574 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784576 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784579 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784583 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784586 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784589 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784599 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784602 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784604 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784607 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784615 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784618 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.788427 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784621 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784624 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784626 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784630 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784632 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784649 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784652 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784656 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784660 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784664 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784667 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784669 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784672 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784674 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784677 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784679 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784682 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784684 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784687 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.788926 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784689 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784692 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784694 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784697 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784704 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784707 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784709 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784712 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784714 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784717 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784719 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784722 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784731 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784734 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784736 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784739 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784742 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784745 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784748 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784750 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.789373 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784754 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784756 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784759 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784761 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784764 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784766 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784769 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784771 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784774 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784776 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784779 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784782 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784784 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784787 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784789 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784791 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784795 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784798 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784800 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784803 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.789872 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784805 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784808 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.784810 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784908 5393 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784915 5393 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784937 5393 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784942 5393 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784946 5393 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784949 5393 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784954 5393 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784958 5393 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784962 5393 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784965 5393 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784969 5393 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784972 5393 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784975 5393 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784978 5393 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784981 5393 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784984 5393 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784986 5393 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784989 5393 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784996 5393 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.784999 5393 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785002 5393 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:03:08.790372 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785005 5393 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785008 5393 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785012 5393 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785015 5393 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785019 5393 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785023 5393 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785026 5393 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785028 5393 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785032 5393 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785035 5393 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785038 5393 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785042 5393 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785045 5393 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785048 5393 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785057 5393 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785068 5393 flags.go:64] FLAG: --enable-server="true" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785071 5393 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785078 5393 flags.go:64] FLAG: --event-burst="100" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785081 5393 flags.go:64] FLAG: --event-qps="50" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785084 5393 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785087 5393 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785090 5393 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785094 5393 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785097 5393 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785100 5393 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:03:08.791358 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785103 5393 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785106 5393 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785109 5393 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785112 5393 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785115 5393 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785118 5393 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785121 5393 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785123 5393 flags.go:64] FLAG: --feature-gates="" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785127 5393 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785130 5393 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785134 5393 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785137 5393 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785140 5393 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785143 5393 flags.go:64] FLAG: --help="false" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785146 5393 flags.go:64] FLAG: --hostname-override="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785151 5393 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785154 5393 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785157 5393 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785160 5393 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785164 5393 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785167 5393 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785169 5393 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785178 5393 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:03:08.791984 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785181 5393 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785184 5393 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785187 5393 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785190 5393 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785193 5393 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785196 5393 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785199 5393 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785202 5393 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785205 5393 flags.go:64] FLAG: --lock-file="" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785208 5393 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785211 5393 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785214 5393 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785219 5393 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785222 5393 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785225 5393 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785228 5393 flags.go:64] FLAG: --logging-format="text" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785231 5393 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785234 5393 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785237 5393 flags.go:64] FLAG: --manifest-url="" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785240 5393 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785245 5393 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785248 5393 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785251 5393 flags.go:64] FLAG: --max-pods="110" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785254 5393 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785259 5393 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:03:08.792548 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785262 5393 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785264 5393 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785268 5393 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785270 5393 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785273 5393 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785280 5393 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785283 5393 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785292 5393 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785295 5393 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785298 5393 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785301 5393 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785304 5393 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785307 5393 flags.go:64] FLAG: --port="10250" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785310 5393 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785313 5393 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ebfd44319ea487b5" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785316 5393 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785319 5393 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785322 5393 flags.go:64] FLAG: --register-node="true" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785325 5393 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785329 5393 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785332 5393 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785334 5393 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785337 5393 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785341 5393 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785344 5393 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785347 5393 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:03:08.793167 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785350 5393 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785353 5393 flags.go:64] FLAG: --runonce="false" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785356 5393 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785359 5393 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785363 5393 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785368 5393 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785371 5393 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785374 5393 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785377 5393 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785380 5393 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785383 5393 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785386 5393 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785389 5393 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785392 5393 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785401 5393 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785404 5393 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785407 5393 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785412 5393 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785415 5393 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785418 5393 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785425 5393 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785427 5393 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785431 5393 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785433 5393 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785436 5393 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:03:08.793786 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785439 5393 flags.go:64] FLAG: --v="2" Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785443 5393 flags.go:64] FLAG: --version="false" Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785447 5393 flags.go:64] FLAG: --vmodule="" Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785451 5393 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.785454 5393 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785664 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785669 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785673 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785676 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785679 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785681 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785684 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785690 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785694 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785697 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785700 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785702 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785705 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785708 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785710 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.794382 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785713 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785716 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785724 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785727 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785730 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785733 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785737 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785740 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785743 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785745 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785748 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785750 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785753 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785756 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785759 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785761 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785764 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785767 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785769 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.794865 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785772 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785775 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785777 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785780 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785783 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785787 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785789 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785792 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785795 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785797 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785800 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785803 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785805 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785808 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785811 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785813 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785821 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785824 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785827 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785831 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.795347 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785834 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785836 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785839 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785841 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785844 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785847 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785849 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785852 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785855 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785857 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785860 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785862 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785865 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785867 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785870 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785873 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785875 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785879 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785882 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785884 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.795837 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785887 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785890 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785892 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785895 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785898 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785900 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785903 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785905 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785908 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785916 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785919 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785923 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785925 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785933 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785936 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785940 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785944 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785947 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785950 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.796318 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785953 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785955 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785958 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785961 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785963 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785965 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785968 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785970 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785973 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785976 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785979 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785982 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.785985 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.786011 5393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786059 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786064 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.796893 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786067 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786072 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786075 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786077 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786080 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786083 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786086 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786089 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786093 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786095 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786098 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786101 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786104 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786106 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786109 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786111 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786114 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786117 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786119 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786122 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.797281 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786124 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786127 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786130 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786132 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786135 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786137 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786140 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786144 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786147 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786149 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786153 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786158 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786162 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786165 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786167 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786170 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786173 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786176 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786179 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.797827 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786181 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786186 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786188 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786191 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786194 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786196 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786199 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786201 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786204 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786206 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786209 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786212 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786214 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786217 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786219 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786223 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786226 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786229 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786232 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.798276 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786234 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786240 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786242 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786245 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786248 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786251 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786253 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786256 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786259 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786261 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786264 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786268 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786271 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786273 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786277 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786280 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786283 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786286 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786288 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.798896 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786291 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786294 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786297 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786299 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786301 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786304 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786307 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786309 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786312 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786314 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786317 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786319 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786322 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786325 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786328 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786331 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786334 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786337 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786339 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.799388 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786342 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786344 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786347 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786349 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786352 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786355 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786357 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.786361 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.790191 5393 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.790206 5393 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.790223 5393 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.790228 5393 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.790477 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791053 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791060 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791065 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791070 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791074 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791079 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791084 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.800079 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791088 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791100 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791105 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791113 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791121 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791128 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791133 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791137 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791142 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791147 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791151 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791155 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791160 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791169 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791174 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791179 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791184 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791188 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791192 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791197 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.800923 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791201 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791205 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791209 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791214 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791218 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791223 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791232 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791236 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791242 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791246 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791250 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791253 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791330 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791482 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791488 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791491 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791494 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791497 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791500 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.801764 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791507 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791510 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791513 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791516 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791519 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791522 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791525 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791528 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791531 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791534 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791537 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791540 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791543 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791545 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791548 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791551 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791554 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791557 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791562 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.802591 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791567 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791570 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791573 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791577 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791579 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791582 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791585 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791587 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791590 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791593 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791595 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791598 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791601 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791604 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791607 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791610 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791612 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791615 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791618 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791620 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.803416 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791623 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791626 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791628 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791631 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791649 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791653 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791657 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791660 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791664 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791667 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791670 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791672 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791675 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791678 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791681 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791683 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791686 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791689 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791691 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.804244 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791694 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.791726 5393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791869 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791875 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791879 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791883 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791886 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791890 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791892 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791895 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791898 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791910 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791913 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791916 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791919 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791921 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.805063 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791924 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791927 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791930 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791933 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791935 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791938 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791941 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791944 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791946 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791949 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791952 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791954 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791957 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791959 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791962 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791964 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791968 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791973 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791976 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.805754 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791980 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791982 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791985 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791988 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791991 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791993 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791996 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.791999 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792001 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792010 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792013 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792016 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792018 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792021 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792024 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792027 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792029 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792032 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792034 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792037 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.806548 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792039 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792042 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792044 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792047 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792049 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792052 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792054 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792057 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792059 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792062 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792064 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792067 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792070 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792074 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792077 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792080 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792083 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792086 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792089 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.807379 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792091 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792094 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792096 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792105 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792108 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792110 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792113 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792115 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792118 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792120 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792123 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792126 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792129 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792131 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792134 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792136 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792139 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792141 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792143 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792146 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.808287 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792149 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792151 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792154 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792156 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792158 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792161 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792163 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792166 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792168 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792171 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792173 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792176 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792178 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792181 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.792203 5393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792254 5393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:08.809202 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792265 5393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792269 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792271 5393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792274 5393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792277 5393 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792279 5393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792283 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792285 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792288 5393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792290 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792294 5393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792298 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792301 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792304 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792306 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792309 5393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792311 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792314 5393 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792316 5393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:08.809892 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792319 5393 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792321 5393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792324 5393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792327 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792329 5393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792341 5393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792344 5393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792347 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792349 5393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792352 5393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792355 5393 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792357 5393 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792360 5393 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792362 5393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792371 5393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792374 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792377 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792380 5393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792382 5393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:08.810729 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792385 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792387 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792390 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792392 5393 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792395 5393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792397 5393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792400 5393 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792404 5393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792408 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792411 5393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792414 5393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792416 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792419 5393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792422 5393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792424 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792427 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792429 5393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792432 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792434 5393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:08.811227 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792437 5393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792440 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792442 5393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792445 5393 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792447 5393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792450 5393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792452 5393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792455 5393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792457 5393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792466 5393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792469 5393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792471 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792474 5393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792476 5393 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792479 5393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792482 5393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792485 5393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792487 5393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792489 5393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792492 5393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:08.811694 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792494 5393 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792497 5393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792499 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792502 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792504 5393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792507 5393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792510 5393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792512 5393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792514 5393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792517 5393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792519 5393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792522 5393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792525 5393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792527 5393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792530 5393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792533 5393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792535 5393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792538 5393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792540 5393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792543 5393 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:08.812167 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792545 5393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792548 5393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792556 5393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792559 5393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792562 5393 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792565 5393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792568 5393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: W1101 20:03:08.792571 5393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.792666 5393 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.793633 5393 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.793705 5393 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.793876 5393 server.go:1014] "Starting client certificate rotation" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.794021 5393 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.794081 5393 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:56:57 +0000 UTC" deadline="2028-08-13 13:13:38.108567368 +0000 UTC" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.794121 5393 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15617h10m29.314448688s" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.794442 5393 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:08.812663 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.795584 5393 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:08.813047 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.796959 5393 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:03:08.813047 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.798864 5393 log.go:25] "Validated CRI v1 image API" Nov 01 20:03:08.813047 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.799590 5393 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:03:08.813047 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.802403 5393 fs.go:135] Filesystem UUIDs: map[498a6f32-2f0b-4c65-b99d-e9484dd74add:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cb440668-8f60-45b8-8567-fce71e3e278c:/dev/nvme0n1p3] Nov 01 20:03:08.813169 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.802429 5393 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~projected/kube-api-access-v8kg5:{mountpoint:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~projected/kube-api-access-v8kg5 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~projected/kube-api-access-hbnqc:{mountpoint:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~projected/kube-api-access-hbnqc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0effeb7a-2a88-48ad-9df6-4cd132b14052/volumes/kubernetes.io~projected/kube-api-access-rpb4w:{mountpoint:/var/lib/kubelet/pods/0effeb7a-2a88-48ad-9df6-4cd132b14052/volumes/kubernetes.io~projected/kube-api-access-rpb4w major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~projected/kube-api-access-dcr84:{mountpoint:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~projected/kube-api-access-dcr84 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~projected/kube-api-access-6wh5x:{mountpoint:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~projected/kube-api-access-6wh5x major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48faedf2-1476-4c1a-842d-e4a6e5ceeb82/volumes/kubernetes.io~projected/kube-api-access-lk569:{mountpoint:/var/lib/kubelet/pods/48faedf2-1476-4c1a-842d-e4a6e5ceeb82/volumes/kubernetes.io~projected/kube-api-access-lk569 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a46610c-94b6-4ec2-a351-e83bf7b95032/volumes/kubernetes.io~projected/kube-api-access-kjhhp:{mountpoint:/var/lib/kubelet/pods/4a46610c-94b6-4ec2-a351-e83bf7b95032/volumes/kubernetes.io~projected/kube-api-access-kjhhp major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~projected/kube-api-access-hkcg8:{mountpoint:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~projected/kube-api-access-hkcg8 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~projected/kube-api-access-ttkdq:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~projected/kube-api-access-ttkdq major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b696755-be38-4546-82e1-9b485bdef642/volumes/kubernetes.io~projected/kube-api-access-pfqjw:{mountpoint:/var/lib/kubelet/pods/8b696755-be38-4546-82e1-9b485bdef642/volumes/kubernetes.io~projected/kube-api-access-pfqjw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902e2a0d-1e93-45e1-87cf-fd428a800f92/volumes/kubernetes.io~projected/kube-api-access-98sd5:{mountpoint:/var/lib/kubelet/pods/902e2a0d-1e93-45e1-87cf-fd428a800f92/volumes/kubernetes.io~projected/kube-api-access-98sd5 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~projected/kube-api-access-55rgc:{mountpoint:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~projected/kube-api-access-55rgc major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~projected/kube-api-access-g96z8:{mountpoint:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~projected/kube-api-access-g96z8 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~secret/metrics-certs major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~projected/kube-api-access-w466g:{mountpoint:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~projected/kube-api-access-w466g major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2d65d12-0f39-4bb4-9ced-cce27246077c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b2d65d12-0f39-4bb4-9ced-cce27246077c/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~projected/kube-api-access-ddjh4:{mountpoint:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~projected/kube-api-access-ddjh4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e/volumes/kubernetes.io~projected/kube-api-access-hcntf:{mountpoint:/var/lib/kubelet/pods/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e/volumes/kubernetes.io~projected/kube-api-access-hcntf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c55e75ab-72d7-43bb-b1ff-e385180a3c70/volumes/kubernetes.io~projected/kube-api-access-k6j2h:{mountpoint:/var/lib/kubelet/pods/c55e75ab-72d7-43bb-b1ff-e385180a3c70/volumes/kubernetes.io~projected/kube-api-access-k6j2h major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~projected/kube-api-access-j658n:{mountpoint:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~projected/kube-api-access-j658n major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~projected/kube-api-access-b4fj4:{mountpoint:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~projected/kube-api-access-b4fj4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffffa9d9-1fea-40cb-85c6-1a1edb49470d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ffffa9d9-1fea-40cb-85c6-1a1edb49470d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:316 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/340213e1dc0d4f8268675acd6b4f180ed15011b6fa62184e0cc112798f69feac/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/21eec1735bedeeeae9e2e48167aa3609272fab3be4d0ba4ddea15ba607f9fa0f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0f85770fa7d68c6df8855ebe6f8c33072b29c22cf4e105fc0ab7dabe0b269a4c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/c6e3a57a9de0a4f7af82eff150c3eceb918d56cfcdbbfd2ba744861f90002c07/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a00bb701652477cb81d71b7cde9ae587ad5a7f9877247c835530cc3017c66ac5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/29207a8730322c165d1301633365b3a63ceefe30f23c8f54711e8e5477afe819/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/73e52df931bc4bee8aaa45fa44c1e20f18fbe98a073a10937e11261e3ef587a3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/935171b91e31794616b2de940a68928cabffa40110e25348f21b75f9a425b410/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/10b663c11bf55ac1c435add09dd2d4105d8b09806b5108f0bda41b5d4e703461/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/c38547b967371bd4fdfe11b9fcabd07db4a99a6a653f0cd40a166bda60c79677/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/4f03a91750834f8479a9068f57e06e25ab94e74e9dac12a9f84b7b3679e1c54b/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3af7cda4a2cbab75e3ffc0ed3d182cb1dd02aa686561f07ca399a0a1f86d5db0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e8515f2522d2ff68416d2aef2cad8234c1c28d63b575d00e5a04fb515e0f7e78/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6ab7baeb1b77e0a2232a68ee66293240133ffc0c683e143a77ec960b96a85d75/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f6e440f52aa5f7a88925ad38e81e7bbb96abf69daf9d523e9be2783d3425d614/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7f78ecc213d158237f491923fbce5294f4a21d5f0ef100ca7d78b1688dc3522a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2f77d536693b529692e8acdd8e46122825fcbf4381168b80a219e872526e80bf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c0b772a76f2509d11c97d30deff5a0ee7f56de8914e502e23b554f916549bfbd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/810c804c63d9397e42bc8e89ecb5e92cba37f0aab8adc111afb056c0e2e5d365/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/586675c09889f39e752add40274fd1059b88a2682aa8a6b84b059e3e0d92f6d4/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/927e7b4cbbdef6e19eee1855b77e5960b01de2269f5d8ffb843f173e03ad2ec6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7c17aed72f7892919188f7967cec646012cf5b032971274ce94ff5b6d091d076/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4458fa112aca8537fbf366c9327a245082fef12f2698526ef0c8557453ae7839/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/72fe0398a782fd94b42ae5f476d05ed577dba91670741c9f7f1305b84849af23/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/f4f8e4b869640272c2a60725d0a7bfdd8fc43109d4024af7f53c19476f97d237/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/c78122cd303ed306eed0c0365fd4ef3190db64f3fbcc2ca27dd6f40f07a9d6e5/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/99357c8f959847d086d4337a5307492d08e694a04aa03b6b636dd25d454179b5/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/649e82c666e5462934e10b3b6a544377e8f774330e848b93c3fd438d849f6acf/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/974dd28e495e16490c4929fba30f55ff731f45ec4b209a67f78220492a122c6a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/0917e48f25c34342c935f2e5dd442fabc761c75c49e98a54076a76a32abd37ee/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/19fc62de49e8af899544a4b100446da36e83441d867a8c4593511221719838d7/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/6df19c3a96948ac11416d9c85cb794d821fc09777e6d30ae87a7c694a14a9f43/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/c27a0e62a5d83dce6bbb5f85f64211ba25b8e53c823abc42d37e2d6ded87d98b/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/c20f8fa53ffe3426a2780618f6c24c27e2cb4e171ac6d016587146cad1937a00/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/eaeb1f29af3306d8d1781f5d9b804ef11efec35affe2d12bd10f3bc85f3fa4a2/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/b8de982c2bd98fb4a84ca6760a81d3ac20583e384913faacb18c28f212ac730f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/164728de19701cce87fc839a426c99e82fddea082bcb977dddd9f4348ca47664/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/f938a58e92d09e349a2c4f6a5a3fb74277ed9453efc84660281fb45426560b67/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/c5945b3d85619c450e24bf40acd99d73d16029cdfc89aef3bdd3bae2ddf8202f/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/61a266377a29ccc1f3e744e2ced0f9921b606e48bd4f5a759a4094cdb1cf817f/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/8ae23b8f950ee94eae623fe2ff1c5aed320f1d61559d14f019f1b8c92bf4fbb4/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/d050250b60e1f064dc9cd59f54e1850b8d9519108ec72d0527d82f17e3cd27aa/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/43ca22d9525e1c9a62ffb6be6ef13412d83baebf1099f598a1cb0b04d77b752e/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/3f8e4a46ca0f2538c138fba5f74a040e02667c0b0f8c508277d17ece80373d7d/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/1c27b99fe67cd124474e1043916ed82ed84d415269c22c4d0350f46be0f0d69a/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/760ea025b8b556b20ff177fdfd1b215b72267ff9cd07db3ad7e32c378299f557/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/1909b8879475f721b07eff0fb71ba1ff3026563bf1d4363330d4c19358061072/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/54f10124d946ff93e67290d964930325a900fa9892ca5d4a41f2b4358ebcbeac/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/7351cb1f8c8c61cb5771ab58d16461893bafc475efaa2422345862ad817079e6/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/240b985e4df4a7b68bd723606a43b278dfaa3c3814065b65c4ef83902540e884/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/b8c88799ddfe5b3f695347f7c2e08495265feb5d1cb1f4b53dda0ef60a412370/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/ad8e71b5165d42eed8b0bf663dbfcb9ec0a4c2967c8af3621a4f171799e4fddb/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/722b6f462a0f37fbd6fd3cd33e5aadf256c1958d5ffbadf2d65ae390e59c6756/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/86ed076c828b1f89668d5ccdf93290bf12f44664fa5a6c0a229bf2e6b2b8e77c/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/4578fea3e1535fef72d95378771795c4f6dad0cd670c1c315a8e4079fc104b0e/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/984a456f4984ecadc73559a7c97a7473c2dc66662f854d96b4f5a476245115f8/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/d06c56b33542d876dd626faf118007fd9a348e7444653b6b15daa8dfe6d663d8/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/fb8b2aac7cbdb13846e5dc4583c120bce7425b07c80dbd71f1e9ae4cb23da028/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/416ce637099bbedd387a310b68b575b80723fb0a0280f1255e9556235bc07c5a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/be302bed46d7b7992fb05971bf67aea8ce8af5a6ebd074f806178d508377c829/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3075de4b5700679794665cdd2be482b2169123f7a6296909579d4bf6649df454/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f8486446616bd1be0ef01127ca8429fdb6ee7feebac29240c29a2acdbdf89b34/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2967edafa138288d4381b1772ea818dbd5410981d1fdb7556eaa61c0d7dad23e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/d549bc2b1fe005dca59a9e46c45a26303d870e789a1dbfde1a0e33dc2730c9ef/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0c1cbfb4ff66883e17361eba80ad3fa1353cfa80e5db01d270f981f819ddce09/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fd8f4148588a622636f204ca811273dbb73f6414a43c7f0858a1c2b9bc7611a5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/557c141f51623940c6da3788bc8c520c1ebfce42065805c6e432b2056c7e7689/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 20:03:08.813489 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.813143 5393 manager.go:217] Machine: {Timestamp:2026-11-01 20:03:08.812702613 +0000 UTC m=+0.061757054 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120319 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af26b6daa089661be53f69db44de2 SystemUUID:ec2af26b-6daa-0896-61be-53f69db44de2 BootID:e67060a4-6958-4af0-8c76-085d4325ee95 Filesystems:[{Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~projected/kube-api-access-6wh5x DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/902e2a0d-1e93-45e1-87cf-fd428a800f92/volumes/kubernetes.io~projected/kube-api-access-98sd5 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e/volumes/kubernetes.io~projected/kube-api-access-hcntf DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~projected/kube-api-access-v8kg5 DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~projected/kube-api-access-j658n DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~projected/kube-api-access-hbnqc DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b2d65d12-0f39-4bb4-9ced-cce27246077c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~projected/kube-api-access-ddjh4 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~projected/kube-api-access-g96z8 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b696755-be38-4546-82e1-9b485bdef642/volumes/kubernetes.io~projected/kube-api-access-pfqjw DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:261 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~projected/kube-api-access-ttkdq DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48faedf2-1476-4c1a-842d-e4a6e5ceeb82/volumes/kubernetes.io~projected/kube-api-access-lk569 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:247 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~projected/kube-api-access-55rgc DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~projected/kube-api-access-b4fj4 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~projected/kube-api-access-w466g DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~projected/kube-api-access-hkcg8 DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c55e75ab-72d7-43bb-b1ff-e385180a3c70/volumes/kubernetes.io~projected/kube-api-access-k6j2h DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~projected/kube-api-access-dcr84 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffffa9d9-1fea-40cb-85c6-1a1edb49470d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0effeb7a-2a88-48ad-9df6-4cd132b14052/volumes/kubernetes.io~projected/kube-api-access-rpb4w DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a46610c-94b6-4ec2-a351-e83bf7b95032/volumes/kubernetes.io~projected/kube-api-access-kjhhp DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1070ff71a35d1cd MacAddress:a6:c7:1c:18:08:8e Speed:10000 Mtu:8901} {Name:12ab677e6a884f7 MacAddress:16:16:7f:66:7f:b9 Speed:10000 Mtu:8901} {Name:31c6793e57ffafc MacAddress:a2:21:d0:22:9e:1d Speed:10000 Mtu:8901} {Name:993de2c9c1272f4 MacAddress:4e:91:57:5c:7e:b0 Speed:10000 Mtu:8901} {Name:9bb7195d0245c67 MacAddress:56:0f:0f:8b:11:36 Speed:10000 Mtu:8901} {Name:afc994dbaf50181 MacAddress:66:a5:48:d1:5b:5f Speed:10000 Mtu:8901} {Name:b2709fa597dae7a MacAddress:66:7f:4f:e9:44:dd Speed:10000 Mtu:8901} {Name:bd72783145cd98e MacAddress:3e:e4:f8:d4:26:33 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:39:0a:0d:7c:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:41:11:ef:98:71 Speed:0 Mtu:8901} {Name:ec7b30b2f6cef23 MacAddress:86:59:0d:33:ac:3a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:39:0a:0d:7c:f1 Speed:0 Mtu:9001} {Name:f0ec98fbdb961e3 MacAddress:0e:75:c4:49:09:a5 Speed:10000 Mtu:8901} {Name:fb8da39585aa3e2 MacAddress:ba:08:13:a7:f1:c3 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:d2:6b:78:64:83:e1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:ff:cc:cc:64:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:03:08.813735 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.813493 5393 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:03:08.813735 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.813538 5393 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:03:08.813916 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.813886 5393 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:03:08.814094 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.813917 5393 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-177.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:03:08.814180 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814097 5393 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:03:08.814180 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814110 5393 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:03:08.814180 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814124 5393 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:03:08.814180 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814151 5393 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:03:08.814480 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814253 5393 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:03:08.814480 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814454 5393 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:03:08.814597 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814585 5393 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:03:08.814667 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814649 5393 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:03:08.814725 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814668 5393 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:03:08.814725 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814683 5393 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:03:08.814725 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814695 5393 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:03:08.814725 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.814706 5393 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:03:08.815322 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.815307 5393 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:03:08.815322 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.815320 5393 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:03:08.815858 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.815838 5393 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:03:08.819136 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819121 5393 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:03:08.819136 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819134 5393 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:03:08.819249 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819234 5393 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 20:03:08.819412 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819402 5393 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:03:08.819443 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819430 5393 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:03:08.819529 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819520 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819535 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819541 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819546 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819552 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819558 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819563 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:03:08.819575 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819571 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:03:08.819849 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819579 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:03:08.819849 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819585 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:03:08.819849 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819599 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:03:08.819849 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819609 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:03:08.819849 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819648 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:03:08.819849 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.819657 5393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:03:08.822975 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.822962 5393 server.go:1263] "Started kubelet" Nov 01 20:03:08.823500 ip-10-0-138-177 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:03:08.823723 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.823680 5393 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:03:08.823800 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.823745 5393 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:03:08.824276 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.824254 5393 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:03:08.824493 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.823111 5393 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:03:08.827527 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.827122 5393 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:03:08.828897 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.828874 5393 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:03:08.829002 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.828943 5393 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:03:08.829118 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.829087 5393 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:03:08.829118 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.829101 5393 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:03:08.829373 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.829184 5393 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:03:08.829373 ip-10-0-138-177 kubenswrapper[5393]: E1101 20:03:08.829236 5393 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:03:08.829525 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.829415 5393 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:58 +0000 UTC" deadline="2028-06-01 03:44:41.536848258 +0000 UTC" Nov 01 20:03:08.829525 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.829433 5393 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13855h41m32.707417945s" Nov 01 20:03:08.831329 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831292 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6e1cad-750a-416c-9a32-81bf6a048e75" volumeName="kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4" seLinuxMountContext="" Nov 01 20:03:08.831410 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831334 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6e1cad-750a-416c-9a32-81bf6a048e75" volumeName="kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert" seLinuxMountContext="" Nov 01 20:03:08.831410 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831349 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides" seLinuxMountContext="" Nov 01 20:03:08.831410 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831362 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume" seLinuxMountContext="" Nov 01 20:03:08.831410 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831375 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config" seLinuxMountContext="" Nov 01 20:03:08.831410 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831386 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48faedf2-1476-4c1a-842d-e4a6e5ceeb82" volumeName="kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569" seLinuxMountContext="" Nov 01 20:03:08.831410 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831399 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c55e75ab-72d7-43bb-b1ff-e385180a3c70" volumeName="kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831413 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831428 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831451 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831463 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831475 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78496de5-ccaa-448e-8683-52ea3c9e5de1" volumeName="kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831488 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831502 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831516 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc73791-d8d2-4a7e-84c2-eb514db3fee1" volumeName="kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831528 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831540 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831551 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831561 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831572 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cec48ca-4b08-41d1-bfad-2df7138f4b90" volumeName="kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831583 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831593 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78496de5-ccaa-448e-8683-52ea3c9e5de1" volumeName="kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831604 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831614 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831623 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831658 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c55e75ab-72d7-43bb-b1ff-e385180a3c70" volumeName="kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831671 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" volumeName="kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret" seLinuxMountContext="" Nov 01 20:03:08.831683 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831683 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831699 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cec48ca-4b08-41d1-bfad-2df7138f4b90" volumeName="kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831710 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831721 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831734 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b696755-be38-4546-82e1-9b485bdef642" volumeName="kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831744 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831756 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4" seLinuxMountContext="" Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831763 5393 factory.go:55] Registering systemd factory Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831781 5393 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.832056 5393 factory.go:153] Registering CRI-O factory Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.832072 5393 factory.go:223] Registration of the crio container factory successfully Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.832134 5393 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.832151 5393 factory.go:103] Registering Raw factory Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.832164 5393 manager.go:1199] Started watching for new ooms in manager Nov 01 20:03:08.832581 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.832467 5393 manager.go:319] Starting recovery of all containers Nov 01 20:03:08.833221 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.831766 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib" seLinuxMountContext="" Nov 01 20:03:08.833221 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833149 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x" seLinuxMountContext="" Nov 01 20:03:08.833221 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833170 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls" seLinuxMountContext="" Nov 01 20:03:08.833221 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833187 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902e2a0d-1e93-45e1-87cf-fd428a800f92" volumeName="kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5" seLinuxMountContext="" Nov 01 20:03:08.833221 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833201 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d65d12-0f39-4bb4-9ced-cce27246077c" volumeName="kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca" seLinuxMountContext="" Nov 01 20:03:08.833221 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833213 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c55e75ab-72d7-43bb-b1ff-e385180a3c70" volumeName="kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833225 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a46610c-94b6-4ec2-a351-e83bf7b95032" volumeName="kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833238 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833249 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833261 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833272 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc73791-d8d2-4a7e-84c2-eb514db3fee1" volumeName="kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833285 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833296 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833309 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833352 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833364 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833377 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cec48ca-4b08-41d1-bfad-2df7138f4b90" volumeName="kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833389 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833401 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833413 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833425 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833436 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833458 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9f1d631-6c85-4c27-b21e-9ddc9c71db4e" volumeName="kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf" seLinuxMountContext="" Nov 01 20:03:08.833463 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833471 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48faedf2-1476-4c1a-842d-e4a6e5ceeb82" volumeName="kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833483 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d65d12-0f39-4bb4-9ced-cce27246077c" volumeName="kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833497 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833509 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833521 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833533 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833552 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833564 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b696755-be38-4546-82e1-9b485bdef642" volumeName="kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833576 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833589 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833601 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833612 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9f1d631-6c85-4c27-b21e-9ddc9c71db4e" volumeName="kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833624 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833752 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833770 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833784 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833797 5393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp" seLinuxMountContext="" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833808 5393 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:03:08.834297 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.833814 5393 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:03:08.862182 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.862135 5393 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:03:08.863819 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.863688 5393 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:03:08.863819 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.863708 5393 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:03:08.863819 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.863730 5393 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:03:08.866016 ip-10-0-138-177 kubenswrapper[5393]: E1101 20:03:08.863773 5393 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 20:03:08.886484 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.886452 5393 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="df41f5056a9e85c1be7e4409891609efd74e2f9ae54c663a4b93d8fcd69ba89a" exitCode=0 Nov 01 20:03:08.886484 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.886481 5393 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="6f082ebaab1f2cf4c76d7cd410f5d32f6022d55e3cda43d1dcd265655d62e533" exitCode=0 Nov 01 20:03:08.886631 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.886491 5393 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="e25c61da0b3b5f6f5f190fa15899ff3f1fd18c1c8f9646ad6c3e65c0e0249eae" exitCode=0 Nov 01 20:03:08.886631 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.886498 5393 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="7af9afa9774b9bedd33ebca95bfc9c001e26c0a6d95dfca4838448ab61b98c1d" exitCode=0 Nov 01 20:03:08.886631 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.886506 5393 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="c45affffaf27c13b174d2d0da7581ba4b76031c250f846d20795e25135652f56" exitCode=0 Nov 01 20:03:08.886631 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.886513 5393 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="3c1afb9ed6c59fb51a39547376e7102edb6ff7ee861a2cc366bcc3207ac7c892" exitCode=0 Nov 01 20:03:08.907588 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.907561 5393 generic.go:356] "Generic (PLEG): container finished" podID="ec2f737d93958347cc920d4e5916d818" containerID="bc6296c1649bd5108fb4124fd3093edae08faad6af36d5c471879182568bb22e" exitCode=0 Nov 01 20:03:08.915495 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.915473 5393 generic.go:356] "Generic (PLEG): container finished" podID="af45e616-0778-4869-9066-7acad174e56a" containerID="9fbfb6015e1e27fc25f2aadbbf492117a9dc927ccd7d39e74e9d4ed288673e5e" exitCode=0 Nov 01 20:03:08.929400 ip-10-0-138-177 kubenswrapper[5393]: E1101 20:03:08.929379 5393 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:03:08.931622 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.931605 5393 manager.go:324] Recovery completed Nov 01 20:03:08.950600 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.950552 5393 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:08.952590 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.952574 5393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:08.952668 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.952603 5393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:08.952668 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.952617 5393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:08.955022 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955006 5393 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:03:08.955022 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955021 5393 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:03:08.955127 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955026 5393 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:08.955127 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955040 5393 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:03:08.955234 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955220 5393 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 20:03:08.955264 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955235 5393 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 20:03:08.955264 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955251 5393 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 20:03:08.955264 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955256 5393 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 20:03:08.955264 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955262 5393 policy_none.go:50] "Start" Nov 01 20:03:08.955368 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955272 5393 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:03:08.955368 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955281 5393 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:03:08.955419 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955402 5393 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 20:03:08.955419 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955409 5393 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 20:03:08.955419 ip-10-0-138-177 kubenswrapper[5393]: I1101 20:03:08.955414 5393 policy_none.go:44] "Start" Nov 01 20:03:08.955495 ip-10-0-138-177 kubenswrapper[5393]: fatal error: concurrent map iteration and map write Nov 01 20:03:08.957602 ip-10-0-138-177 kubenswrapper[5393]: goroutine 815 [running]: Nov 01 20:03:08.957652 ip-10-0-138-177 kubenswrapper[5393]: internal/runtime/maps.fatal({0x2633935?, 0x48048f?}) Nov 01 20:03:08.957652 ip-10-0-138-177 kubenswrapper[5393]: runtime/panic.go:1046 +0x18 Nov 01 20:03:08.957652 ip-10-0-138-177 kubenswrapper[5393]: internal/runtime/maps.(*Iter).Next(0xc0013f93a0?) Nov 01 20:03:08.957652 ip-10-0-138-177 kubenswrapper[5393]: internal/runtime/maps/table.go:792 +0x86 Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm.(*containerManagerImpl).GetNodeAllocatableReservation(0xc000658608) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/node_container_manager_linux.go:281 +0xb6 Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).defaultNodeStatusFuncs.MachineInfo.func1({0x29f8ef0?, 0x3c3a320?}, 0xc00137e008) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/nodestatus/setters.go:305 +0x692 Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).setNodeStatus(0xc00036fc08, {0x29f8ef0, 0x3c3a320}, 0xc00137e008) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:714 +0x28a Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initialNode(0xc00036fc08, {0x29f8ef0, 0x3c3a320}) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:411 +0x9ab Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).getNodeAnyWay(0xc00036fc08) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:305 +0x35 Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin.(*DRAPluginManager).wipeResourceSlices.func1({0x29f8e10, 0xc00016e450}) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin/dra_plugin_manager.go:231 +0x82 Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x29f8e10?, 0xc00016e450?}, 0xc00016e450?) Nov 01 20:03:08.957775 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:159 +0x62 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x29f8e10, 0xc00016e450}, {0x3b9aca00, 0x4000000000000000, 0x3fc999999999999a, 0x64, 0x45d964b800}, 0xc0013f9f08) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:501 +0xae Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin.(*DRAPluginManager).wipeResourceSlices(0xc000b701b0, {0x29f8e10, 0xc00016e450}, {0x0, 0x0}) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin/dra_plugin_manager.go:230 +0x150 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin.NewDRAPluginManager.func2() Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin/dra_plugin_manager.go:178 +0xf9 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin.NewDRAPluginManager in goroutine 549 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/plugin/dra_plugin_manager.go:172 +0x2c5 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: goroutine 1 [select]: Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app.run({0x29f8e48, 0xc000964370}, 0xc000101008, 0xc0005bfa00, {0xf?, 0xc000880d20?}) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:925 +0x20c5 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app.Run({0x29f8e48, 0xc000964370}, 0xc000101008, 0xc0005bfa00, {0x7fbfb16fac88, 0xc000482080}) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:532 +0x331 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc0004e2f08, {0xc000052250, 0x11, 0x11}) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:295 +0xc8c Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: github.com/spf13/cobra.(*Command).execute(0xc0004e2f08, {0xc000052250, 0x11, 0x11}) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: github.com/spf13/cobra@v1.10.0/command.go:1015 +0xb02 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: github.com/spf13/cobra.(*Command).ExecuteC(0xc0004e2f08) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: github.com/spf13/cobra@v1.10.0/command.go:1148 +0x465 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: github.com/spf13/cobra.(*Command).Execute(...) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: github.com/spf13/cobra@v1.10.0/command.go:1071 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-base/cli.run(0xc0004e2f08) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-base/cli/run.go:146 +0x23e Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-base/cli.Run(0x29f8dd8?) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-base/cli/run.go:44 +0x17 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: main.main() Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/kubelet.go:37 +0x2a Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: goroutine 161 [select]: Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue.(*Typed[...]).updateUnfinishedWorkLoop(0x2a10ba0) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/queue.go:363 +0xf8 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: sync.(*WaitGroup).Go.func1() Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:239 +0x4a Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: created by sync.(*WaitGroup).Go in goroutine 1 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:237 +0x73 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: goroutine 160 [sync.WaitGroup.Wait]: Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_SemacquireWaitGroup(0x45321f?, 0xf8?) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:114 +0x2e Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: sync.(*WaitGroup).Wait(0xc000688020) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:206 +0x85 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run(0xc0000007e0) Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:496 +0x2f7 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/certificate.(*manager).Start in goroutine 1 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:408 +0x4f Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: goroutine 196 [syscall]: Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: os/signal.signal_recv() Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: runtime/sigqueue.go:152 +0x29 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: os/signal.loop() Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: os/signal/signal_unix.go:23 +0x13 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: created by os/signal.Notify.func1.1 in goroutine 1 Nov 01 20:03:08.958168 ip-10-0-138-177 kubenswrapper[5393]: os/signal/signal.go:152 +0x1f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 159 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc00009ff50, {0x29d9080, 0xc0002ac000}, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000000000?, {0x29d9080?, 0xc0002ac000?}, 0xd8?, 0x8c81ae?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000a89140, 0x2540be400, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/certificate.addCertRotation in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/certificate/transport.go:156 +0x379 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 533 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f8e10, 0xc000d5e180}, 0xc000aab560, {0x29d9080, 0xc000aab590}, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f8e10, 0xc000d5e180}, 0xc000aab560, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run.func4() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:493 +0x6a Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/certificate.(*manager).run in goroutine 399 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:491 +0x2e5 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 182 [IO wait]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb163ca00, 0x72) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc000b2a000?, 0xc000dfa000?, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc000b2a000, {0xc000dfa000, 0x2000, 0x2000}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).read(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os/file_posix.go:30 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).Read(0xc0009ac048, {0xc000dfa000?, 0x3b?, 0x0?}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:144 +0x4f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: github.com/euank/go-kmsg-parser/kmsgparser.(*parser).Parse.func1() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: github.com/euank/go-kmsg-parser@v2.0.0+incompatible/kmsgparser/kmsgparser.go:129 +0xb1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/euank/go-kmsg-parser/kmsgparser.(*parser).Parse in goroutine 56 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: github.com/euank/go-kmsg-parser@v2.0.0+incompatible/kmsgparser/kmsgparser.go:123 +0x78 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 112 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/klog/v2.(*flushDaemon).run.func1() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/klog/v2@v2.130.1/klog.go:1141 +0xff Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/klog/v2.(*flushDaemon).run in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/klog/v2@v2.130.1/klog.go:1137 +0x171 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 211 [chan receive]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server.setupSignalContext.func1() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/signal.go:66 +0x31 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apiserver/pkg/server.setupSignalContext in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/signal.go:65 +0xe5 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 133 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run.func1({0x29f8e10, 0xc0006666f0}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:436 +0x229 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f8e10?, 0xc0006666f0?}, 0xc00064a030?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f8e10, 0xc0006666f0}, 0xc00064a000, {0x29d9080, 0xc00064a030}, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f8e10, 0xc0006666f0}, 0xc00064a000, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run.func2() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:469 +0x6a Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/certificate.(*manager).run in goroutine 160 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:467 +0x1d9 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 226 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue.(*delayingType[...]).waitingLoop(0x2a0ad60, {{0x2a02068, 0xc0006d0500?}, 0x0?}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/delaying_queue.go:320 +0x385 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/workqueue.newDelayingQueue[...] in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/delaying_queue.go:157 +0x272 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 229 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1.1() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:105 +0x67 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1 in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:104 +0xac Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 230 [chan receive]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc0008d48a0, {0x29f8e48, 0xc0002d9270}, 0x0?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:174 +0x345 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/cmd/kubelet/app.BuildAuthn.func1 in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/auth.go:112 +0x110 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 231 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00097e530, {0x29f8e48, 0xc0002d92c0}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:88 +0x10c Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:52 +0x11a Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 232 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00097e560, {0x29f8e48, 0xc0002d9310}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:88 +0x10c Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:52 +0x11a Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 233 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00097e590, {0x29f8e48, 0xc0002d9360}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:88 +0x10c Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:52 +0x11a Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 234 [sync.Cond.Wait]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_notifyListWait(0xc0005b1710, 0x0) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:606 +0x159 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Cond).Wait(0x0?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: sync/cond.go:71 +0x73 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue.(*Typed[...]).Get(0x2a10ba0) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/queue.go:269 +0xab Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).processNextWorkItem(0xc0008d48a0) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:229 +0x44 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).runWorker(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:224 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x42d5fc?, 0x3c3cb60?}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0006b42a0?}, 0x41af8f?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0006b42a0}, 0xc000764f50, {0x29d9080, 0xc000512000}, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1350540?, {0x29d9080?, 0xc000512000?}, 0x78?, 0xc000b04f70?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00097e5c0, 0x3b9aca00, 0x0, 0x1, 0xc0006b42a0) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run in goroutine 230 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:165 +0x253 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 235 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).watchCAFile(0xc0008d48a0, 0xc0006b42a0) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:194 +0x2f4 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run.func1() Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:169 +0x32 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x0?, 0x0?}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0006b42a0?}, 0x41aed4?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0006b42a0}, 0xc000523f50, {0x29d9080, 0xc0008f7d40}, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc0008f7d40?}, 0x0?, 0x0?) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008f5560, 0xdf8475800, 0x0, 0x1, 0xc0006b42a0) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run in goroutine 230 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:168 +0x327 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 236 [IO wait]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bba200, 0x72) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc0008fc2a0?, 0xc000857e53?, 0x1) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc0008fc2a0, {0xc000857e53, 0x10000, 0x10000}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).read(...) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os/file_posix.go:30 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).Read(0xc00052fb30, {0xc000857e53?, 0x0?, 0x0?}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:144 +0x4f Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify.(*inotify).readEvents(0xc00059f400) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:357 +0xcf Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/fsnotify/fsnotify.newBackend in goroutine 235 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:155 +0x1f6 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 213 [select]: Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00051bcb0, {0x29f8e48, 0xc0002fa730}) Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:88 +0x10c Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1 Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:52 +0x11a Nov 01 20:03:08.959545 ip-10-0-138-177 kubenswrapper[5393]: goroutine 214 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00051bce0, {0x29f8e48, 0xc0002fa780}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:88 +0x10c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:52 +0x11a Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 135 [IO wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bba000, 0x72) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc00061de00?, 0xc00076a000?, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc00061de00, {0xc00076a000, 0x8000, 0x8000}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).Read(0xc00061de00, {0xc00076a000?, 0xc0002a8000?, 0x30?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/fd_posix.go:68 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net.(*conn).Read(0xc00038c1f0, {0xc00076a000?, 0x47d1c5?, 0xc00090dd48?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/net.go:196 +0x45 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio.(*Reader).Read(0xc000880660, {0xc00028e3c4, 0x9, 0x237f660?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio/bufio.go:245 +0x197 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io.ReadAtLeast({0x29d8540, 0xc000880660}, {0xc00028e3c4, 0x9, 0x9}, 0x9) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io/io.go:335 +0x8e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io.ReadFull(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io/io.go:354 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.readFrameHeader({0xc00028e3c4, 0x9, 0x0?}, {0x29d8540?, 0xc000880660?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:250 +0x65 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc00028e380) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:513 +0x6b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc00028e380) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:572 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0002aa008, 0xc0003d4460) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:1640 +0x1c9 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 197 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:414 +0x1dfe Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 212 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000634ec0, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/controlbuf.go:412 +0x108 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001d7080) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/controlbuf.go:575 +0x78 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:472 +0xd2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 197 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:470 +0x23eb Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 215 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00051bd10, {0x29f8e48, 0xc0002fa870}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:88 +0x10c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/grpcsync/callback_serializer.go:52 +0x11a Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 54 [syscall]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall.Syscall(0x0, 0x18, 0xc000fc1f2e, 0x10000) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall/syscall_linux.go:74 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall.read(0x0?, {0xc000fc1f2e?, 0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall/zsyscall_linux_amd64.go:736 +0x38 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall.Read(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall/syscall_unix.go:183 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/utils/inotify.(*Watcher).readEvents(0xc0008be400) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/utils@v0.0.0-20251002143259-bc988d571ff4/inotify/inotify_linux.go:144 +0x69 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/utils/inotify.NewWatcher in goroutine 549 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/utils@v0.0.0-20251002143259-bc988d571ff4/inotify/inotify_linux.go:56 +0x1b6 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 55 [syscall]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall.Syscall(0x0, 0x19, 0xc000de1f2e, 0x10000) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall/syscall_linux.go:74 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall.read(0x0?, {0xc000de1f2e?, 0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall/zsyscall_linux_amd64.go:736 +0x38 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall.Read(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: syscall/syscall_unix.go:183 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/utils/inotify.(*Watcher).readEvents(0xc0008be480) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/utils@v0.0.0-20251002143259-bc988d571ff4/inotify/inotify_linux.go:144 +0x69 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/utils/inotify.NewWatcher in goroutine 549 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/utils@v0.0.0-20251002143259-bc988d571ff4/inotify/inotify_linux.go:56 +0x1b6 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 52 [IO wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bb9e00, 0x72) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc00099a080?, 0xc0009b2000?, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc00099a080, {0xc0009b2000, 0x8000, 0x8000}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).Read(0xc00099a080, {0xc0009b2000?, 0xc0009c6000?, 0x30?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/fd_posix.go:68 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net.(*conn).Read(0xc0009ac000, {0xc0009b2000?, 0x47d1c5?, 0xc000760d48?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/net.go:196 +0x45 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio.(*Reader).Read(0xc0009ae000, {0xc0009c4044, 0x9, 0x237f660?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio/bufio.go:245 +0x197 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io.ReadAtLeast({0x29d8540, 0xc0009ae000}, {0xc0009c4044, 0x9, 0x9}, 0x9) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io/io.go:335 +0x8e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io.ReadFull(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: io/io.go:354 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.readFrameHeader({0xc0009c4044, 0x9, 0x0?}, {0x29d8540?, 0xc0009ae000?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:250 +0x65 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0009c4000) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:513 +0x6b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0009c4000) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:572 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0009c8008, 0xc0001a0b60) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:1640 +0x1c9 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 216 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:414 +0x1dfe Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 53 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0009a0040, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/controlbuf.go:412 +0x108 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000768100) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/controlbuf.go:575 +0x78 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc/internal/transport.NewHTTP2Client.func6() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:472 +0xd2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 216 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/internal/transport/http2_client.go:470 +0x23eb Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 375 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:407 +0xfe Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:404 +0xb2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 218 [IO wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bb9c00, 0x72) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc000888f80?, 0xc000935000?, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc000888f80, {0xc000935000, 0x1000, 0x1000}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).Read(0xc000888f80, {0xc000935000?, 0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/fd_posix.go:68 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net.(*conn).Read(0xc00008f150, {0xc000935000?, 0x408a25?, 0xc000924990?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/net.go:196 +0x45 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*persistConn).Read(0xc0008738c0, {0xc000935000?, 0x872e65?, 0x29d6e20?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:2125 +0x47 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio.(*Reader).fill(0xc00087e960) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio/bufio.go:113 +0x103 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio.(*Reader).Peek(0xc00087e960, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: bufio/bufio.go:152 +0x53 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*persistConn).readLoop(0xc0008738c0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:2278 +0x172 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by net/http.(*Transport).dialConn in goroutine 217 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1947 +0x174f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 219 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*persistConn).writeLoop(0xc0008738c0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:2600 +0xe7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by net/http.(*Transport).dialConn in goroutine 217 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1948 +0x17a5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 237 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002f50e0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/watch/mux.go:268 +0x66 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 238 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record.NewBroadcaster.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:219 +0x2c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/record.NewBroadcaster in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:218 +0x1e5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 376 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc000cbb900, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc000448480?, 0xc00145b6c0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc000cbb900, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc000cbb900, {0x20?, 0x31?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc0006828c0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc000cbb7c0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc000cf5b30?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc000365c80, 0xc000cbb680) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc000cbb680, {0x29dc1a0, 0xc000365c80}, {0x41ae01?, 0x7fbfb81b2ed0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc0007449f0, 0xc000cbb680, {0x2239a40?, 0xc000da3750?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc0007449f0, 0xc000cbb680) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).request(0xc000766780, {0x29f8dd8, 0x3c3a320}, 0xc000da3b00) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1088 +0x3c5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Do(0xc000766780, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1136 +0x127 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/kubernetes/typed/core/v1.(*events).CreateWithEventNamespaceWithContext(0xc00052e058, {0x29f8dd8, 0x3c3a320}, 0xc00091b688) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:85 +0x125 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/kubernetes/typed/core/v1.(*events).CreateWithEventNamespace(0x134cc54?, 0xc0008f4040?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:69 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/kubernetes/typed/core/v1.(*EventSinkImpl).Create(0x41aed4?, 0xc00094a380?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:217 +0x1b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record.recordEvent({0x29f8e48, 0xc0002f53b0}, {0x29f0600?, 0xc00097e000?}, 0xc00091b688, {0x0?, 0x3?, 0x10?}, 0x1?, 0xc00038e000) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:340 +0x17c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record.(*eventBroadcasterImpl).recordToSink(0xc000889800, {0x29f0600, 0xc00097e000}, 0xc000d52288, 0xc00038e000) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:302 +0x158 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartRecordingToSink.func1(0xc0000897a8?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:279 +0x29 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:418 +0x126 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/record/event.go:404 +0xb2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 380 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm.NewContainerManager.func1({0x25ebd43, 0xd}, 0xc0005b2620) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:383 +0x134 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/cm.NewContainerManager in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:381 +0x1572 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 381 [sync.WaitGroup.Wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_SemacquireWaitGroup(0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:114 +0x2e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.(*WaitGroup).Wait(0xc000980810) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:206 +0x85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm.NewContainerManager.func2() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:391 +0x25 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/cm.NewContainerManager in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:390 +0x171f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 382 [sync.Cond.Wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_notifyListWait(0xc0006fa868, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:606 +0x159 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Cond).Wait(0x25f48e0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync/cond.go:71 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*RealFIFO).PopBatch(0xc0006fa840, 0xc00073c030) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/the_real_fifo.go:279 +0x8b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0006fa8f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:234 +0x162 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0xc00016e330?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000527dc8, {0x29d9080, 0xc00016e330}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000527dc8, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext(0xc0006fa8f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:199 +0x41e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext(0xc00096c0b0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:579 +0x39a Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:526 +0x1d Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:161 +0x56 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:159 +0x1df Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 239 [IO wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bb9a00, 0x72) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc0008fdaa0?, 0xc000c3de53?, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc0008fdaa0, {0xc000c3de53, 0x10000, 0x10000}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).read(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os/file_posix.go:30 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).Read(0xc00008f410, {0xc000c3de53?, 0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:144 +0x4f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify.(*inotify).readEvents(0xc00093a640) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:357 +0xcf Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/fsnotify/fsnotify.newBackend in goroutine 385 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:155 +0x1f6 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 137 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0002b9040, {0x29f8e48, 0xc00067ab90}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:891 +0x65 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext.(*Group).StartWithContext.func4() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 382 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 138 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:167 +0x2c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/cache.(*controller).RunWithContext in goroutine 382 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:166 +0xdf Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 383 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*mux).listen(0xc0002ac3c0, {0x29f8ef0, 0x3c3a320}, {0x25dbf0a, 0x4}, 0xc0000c37a0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/mux.go:79 +0x74 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*mux).ChannelWithContext.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/mux.go:73 +0x2b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x42d5fc?, 0x3c3cb60?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0x0?}, 0x41af8f?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0x0}, 0xc000af5f50, {0x29d9080, 0xc000998090}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000998090?}, 0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008ea980, 0x0, 0x0, 0x1, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/config.(*mux).ChannelWithContext in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/mux.go:73 +0x233 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 384 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*sourceFile).run.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/file.go:102 +0x190 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/config.(*sourceFile).run in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/file.go:96 +0xb0 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 385 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*sourceFile).doWatch(0xc0002b90e0, {{0x2a02068?, 0xc0006d0500?}, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/file_linux.go:92 +0x413 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*sourceFile).startWatch.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/file_linux.go:60 +0xae Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x3fc771ddf224b229?, 0x3f8f628c73d4f133?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41aed4?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000ae3f50, {0x29d9080, 0xc000aff890}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x13d91868ef?, {0x29d9080?, 0xc000aff890?}, 0xc0?, 0xc000075208?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008ea9c0, 0x3b9aca00, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Forever(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:80 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/config.(*sourceFile).startWatch in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/file_linux.go:55 +0x11f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 386 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*mux).listen(0xc0002ac3c0, {0x29f8ef0, 0x3c3a320}, {0x25db8fc, 0x3}, 0xc0000c39d0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/mux.go:79 +0x74 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.(*mux).ChannelWithContext.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/mux.go:73 +0x2b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0x0?}, 0x41aed4?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0x0}, 0xc00008df50, {0x29d9080, 0xc000aff860}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000aff860?}, 0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0008eaa40, 0x0, 0x0, 0x1, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/config.(*mux).ChannelWithContext in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/mux.go:73 +0x233 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 139 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc00049a3c0, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc0000dac80?, 0xc00145b6c0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc00049a3c0, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc00049a3c0, {0xc0?, 0xd?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc00067af50?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc00049a3c0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc00016e810?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0004d1b00, 0xc00049a280) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc00049a280, {0x29dc1a0, 0xc0004d1b00}, {0x41ae01?, 0x7fbfb81f8340?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000666ab0, 0xc00049a280, {0x2587380?, 0xc0006513f0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000666ab0, 0xc00049a280) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Watch(0xc00087b680, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:794 +0x1a5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Watch(0xc0006517a8, {0x29f9198?, 0xc0000c2540}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0xc000c80000, ...}, ...}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:201 +0x155 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/core/v1.NewFilteredNodeInformer.func4({0x29f9198, 0xc0000c2540}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/core/v1/node.go:82 +0x12b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*ListWatch).WatchWithContext(0xc?, {0x29f9198?, 0xc0000c2540?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/listwatch.go:309 +0x5d Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).watchList(0xc0009ca0f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:771 +0x528 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).ListAndWatchWithContext(0xc0009ca0f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:423 +0x1ff Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:369 +0x2e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41aed4?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000651ea8, {0x29d90a0, 0xc00067acd0}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2a02068?, {0x29d90a0?, 0xc00067acd0?}, 0x96?, 0x12?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext(0xc0009ca0f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:368 +0x22b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 382 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 387 [sleep]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: time.Sleep(0x3b9aca00) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/time.go:363 +0x165 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config.NewSourceApiserver.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:49 +0x54 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/config.NewSourceApiserver in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:43 +0x1da Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 388 [sync.Cond.Wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_notifyListWait(0xc00041ce98, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:606 +0x159 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Cond).Wait(0x25f48e0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync/cond.go:71 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*RealFIFO).PopBatch(0xc00041ce70, 0xc0003860d0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/the_real_fifo.go:279 +0x8b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00041d550, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:234 +0x162 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0xc0009fe000?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000adfdc8, {0x29d9080, 0xc0009fe000}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000adfdc8, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext(0xc00041d550, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:199 +0x41e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext(0xc00096c160, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:579 +0x39a Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:526 +0x1d Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:161 +0x56 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:159 +0x1df Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 164 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0002b9130, {0x29f8e48, 0xc0003020a0}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:891 +0x65 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext.(*Group).StartWithContext.func4() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 388 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 165 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:167 +0x2c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/cache.(*controller).RunWithContext in goroutine 388 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:166 +0xdf Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 166 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc000686a00, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc000945440?, 0xc00145b6c0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc000686a00, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc000686a00, {0xb0?, 0xad?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc0003022d0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc000686a00) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc0009fe630?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0004d1b00, 0xc000686780) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc000686780, {0x29dc1a0, 0xc0004d1b00}, {0x41ae01?, 0x7fbfb81b1cb0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000666ab0, 0xc000686780, {0x2587380?, 0xc00090b3e0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000666ab0, 0xc000686780) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Watch(0xc0005bc780, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:794 +0x1a5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Watch(0xe, {0x29f9198?, 0xc0000c2540}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0xc000986000, ...}, ...}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:201 +0x155 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/core/v1.NewFilteredServiceInformer.func4({0x29f9198, 0xc0000c2540}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/core/v1/service.go:83 +0x16b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*ListWatch).WatchWithContext(0x0?, {0x29f9198?, 0xc0000c2540?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/listwatch.go:309 +0x5d Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).watchList(0xc0001121e0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:771 +0x528 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).ListAndWatchWithContext(0xc0001121e0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:423 +0x1ff Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:369 +0x2e Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41aed4?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc00090bea8, {0x29d90a0, 0xc0003020f0}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2a02068?, {0x29d90a0?, 0xc0003020f0?}, 0x96?, 0x12?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext(0xc0001121e0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:368 +0x22b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 388 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 140 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: context.(*cancelCtx).propagateCancel.func2() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: context/context.go:523 +0x98 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by context.(*cancelCtx).propagateCancel in goroutine 139 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: context/context.go:522 +0x409 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 923 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00170ad88, 0xb5?, 0x5f5e100) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00170ad88) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 906 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc00087b800, 0xc00093a3c0, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc00087b800, 0x2a06a70?, 0xc000d31508?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 551 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 167 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: context.(*cancelCtx).propagateCancel.func2() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: context/context.go:523 +0x98 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by context.(*cancelCtx).propagateCancel in goroutine 166 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: context/context.go:522 +0x409 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 905 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc0007cfc80, 0xc001854640, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc0007cfc80, 0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 802 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 389 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000af7f50, {0x29d9080, 0xc000631320}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000631320?}, 0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00097e7f0, 0xdf8475800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/util/manager.NewObjectCache in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager/watch_based_manager.go:207 +0x207 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 390 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager.(*objectCache).shutdownWhenStopped(0xc0008d67e0, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager/watch_based_manager.go:378 +0x3b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/util/manager.NewObjectCache in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager/watch_based_manager.go:208 +0x255 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 391 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000b80750, {0x29d9080, 0xc0009fe960}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc0009fe960?}, 0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00097e810, 0xdf8475800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/util/manager.NewObjectCache in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager/watch_based_manager.go:207 +0x207 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 392 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager.(*objectCache).shutdownWhenStopped(0xc0008d6870, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager/watch_based_manager.go:378 +0x3b Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/util/manager.NewObjectCache in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/util/manager/watch_based_manager.go:208 +0x255 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 532 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run.func1({0x29f8e10, 0xc000d5e180}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:436 +0x229 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f8e10?, 0xc000d5e180?}, 0xc000dc0d50?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f8e10, 0xc000d5e180}, 0xc000aab530, {0x29d9080, 0xc000dc0d50}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f8e10, 0xc000d5e180}, 0xc000aab530, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run.func2() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:469 +0x6a Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/certificate.(*manager).run in goroutine 399 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:467 +0x1d9 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 997 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00145c6c8, 0xaf?, 0x5f5e100) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00145c6c8) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 393 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue.(*Typed[...]).updateUnfinishedWorkLoop(0x2a10ba0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/queue.go:363 +0xf8 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync.(*WaitGroup).Go.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:239 +0x4a Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by sync.(*WaitGroup).Go in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:237 +0x73 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 394 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue.(*delayingType[...]).waitingLoop(0x2a0ad60, {{0x2a02068, 0xc0006d0500?}, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/delaying_queue.go:320 +0x385 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/workqueue.newDelayingQueue[...] in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/workqueue/delaying_queue.go:157 +0x272 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 395 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000b82750, {0x29d9080, 0xc000666720}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000666720?}, 0x0?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0004c9d00, 0xdf8475800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Forever(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:80 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/token.NewManager in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/token/token_manager.go:80 +0x1e5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 174 [chan receive]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.newPodContainerDeletor.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/pod_container_deletor.go:51 +0x70 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0xc000002380?, 0x412940?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41aed4?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000ae1f50, {0x29d9080, 0xc000cf5200}, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xd4be407100000000?, {0x29d9080?, 0xc000cf5200?}, 0x58?, 0x0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00099ff00, 0x0, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.newPodContainerDeletor in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/pod_container_deletor.go:49 +0xea Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 402 [IO wait]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bb9600, 0x72) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc0008e6120?, 0xc000d8fe53?, 0x1) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc0008e6120, {0xc000d8fe53, 0x10000, 0x10000}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).read(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os/file_posix.go:30 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).Read(0xc0009ac3f8, {0xc000d8fe53?, 0x0?, 0x0?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:144 +0x4f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify.(*inotify).readEvents(0xc000d0c500) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:357 +0xcf Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/fsnotify/fsnotify.newBackend in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: github.com/fsnotify/fsnotify@v1.9.0/backend_inotify.go:155 +0x1f6 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 403 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/filesystem.(*fsnotifyWatcher).Run.func1() Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/filesystem/watcher.go:81 +0xef Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/util/filesystem.(*fsnotifyWatcher).Run in goroutine 1 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/filesystem/watcher.go:78 +0x4f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: goroutine 404 [select]: Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc000cbb540, 0x0) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc00091e180?, 0xc00145b6c0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc000cbb540, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc000cbb540, {0xa0?, 0x30?}) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc0001d2a00?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc000cbb540) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc000cf54d0?) Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.964145 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0004d1b00, 0xc000cbb400) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc000cbb400, {0x29dc1a0, 0xc0004d1b00}, {0x41ae01?, 0x7fbfb81b2ed0?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000666ab0, 0xc000cbb400, {0x2587380?, 0xc0006536d0?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000666ab0, 0xc000cbb400) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).request(0xc000220600, {0x29f8ef0, 0x3c3a320}, 0xc000653a80) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1088 +0x3c5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Do(0xc000220600, {0x29f8ef0, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1136 +0x127 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Get(0x2a18480, {0x29f8ef0, 0x3c3a320}, {0x7ffdae46ea6b, 0x2a}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x29c75a8, ...}}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:165 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi.waitForAPIServerForever.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi/csi_plugin.go:985 +0x9a Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite.ConditionFunc.WithContext.func1({0x0?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:113 +0x13 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x29f8dd8?, 0x3c3a320?}, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:159 +0x62 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.poll({0x29f8dd8, 0x3c3a320}, 0x0?, 0xc000653dc0, 0xc000653e08) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:245 +0x32 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollImmediateInfiniteWithContext({0x29f8dd8?, 0x3c3a320?}, 0x0?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:229 +0x45 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite(0x0?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:214 +0x3a Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi.waitForAPIServerForever({0x2a2bd58?, 0xc000504c40?}, {0x7ffdae46ea6b?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi/csi_plugin.go:980 +0xdf Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi.initializeCSINode.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi/csi_plugin.go:381 +0x8d Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/volume/csi.initializeCSINode in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/csi/csi_plugin.go:376 +0x132 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 399 [sync.WaitGroup.Wait]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_SemacquireWaitGroup(0x45321f?, 0xf8?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:114 +0x2e Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.(*WaitGroup).Wait(0xc0003bfa10) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync/waitgroup.go:206 +0x85 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate.(*manager).run(0xc000000620) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:496 +0x2f7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/util/certificate.(*manager).Start in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/util/certificate/certificate_manager.go:408 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 413 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000f8cf50, {0x29d9080, 0xc000960540}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000000000?, {0x29d9080?, 0xc000960540?}, 0x18?, 0x9104f1?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006a3890, 0xdf8475800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).StartGarbageCollection in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1645 +0xed Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 397 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc001686f50, {0x29d9080, 0xc000dc0000}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000dc0000?}, 0x0?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000abc900, 0x45d964b800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/images.(*realImageGCManager).Start in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/images/image_gc_manager.go:220 +0x111 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 398 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000c12f50, {0x29d9080, 0xc0009d5440}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006b4310?, {0x29d9080?, 0xc0009d5440?}, 0x80?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000abc940, 0x6fc23ac00, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/images.(*realImageGCManager).Start in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/images/image_gc_manager.go:228 +0x1e7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 414 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc001687f50, {0x29d9080, 0xc000667050}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000000000?, {0x29d9080?, 0xc000667050?}, 0x78?, 0x9104f1?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000d5f4a0, 0x45d964b800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).StartGarbageCollection in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1671 +0x267 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 415 [sleep]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: time.Sleep(0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/time.go:363 +0x165 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).syncLoop(0xc00036fc08, {0x29f8dd8, 0x3c3a320}, 0xc0005b2930, {0x2a01f48, 0xc00036fc08}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:2547 +0x235 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run(0xc00036fc08, 0xc0005b2930) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1909 +0x8db Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/cmd/kubelet/app.startKubelet in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:1270 +0xa9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 416 [IO wait]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb8ba0800, 0x72) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc000769300?, 0x0?, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Accept(0xc000769300) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:613 +0x28c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).accept(0xc000769300) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/fd_unix.go:161 +0x29 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*TCPListener).accept(0xc000635d00) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/tcpsock_posix.go:159 +0x1b Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*TCPListener).Accept(0xc000635d00) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/tcpsock.go:380 +0x30 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*listener).Accept(0xc000600510) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/tls.go:79 +0x24 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Server).Serve(0xc00047a900, {0x29f0c90, 0xc000600510}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/server.go:3463 +0x30c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Server).ServeTLS(0xc00047a900, {0x29f0cf0, 0xc000635d00}, {0x0, 0x0}, {0x0, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/server.go:3532 +0x3a6 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Server).ListenAndServeTLS(0xc00047a900, {0x0, 0x0}, {0x0, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/server.go:3748 +0x115 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/server.ListenAndServeKubeletServer({0x29f8e48, 0xc000964370}, {0x2a1fb58, 0xc00036fc08}, {0x29fa198, 0xc00038f8c0}, {0xc000062020, 0x1, 0x1}, {0x0, ...}, ...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/server/server.go:200 +0x4d6 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).ListenAndServe(0xc00036fc08, {0x29f8e48, 0xc000964370}, 0xc000101140, 0xc000960ff0, {0x29f0660, 0xc0008f7b00}, {0x29e4a60, 0xc00097f260}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3188 +0x116 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/cmd/kubelet/app.startKubelet in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:1274 +0x1c5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 417 [IO wait]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb8ba0a00, 0x72) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc00035c080?, 0x48243e?, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Accept(0xc00035c080) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:613 +0x28c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).accept(0xc00035c080) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/fd_unix.go:161 +0x29 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*UnixListener).accept(0xc000fb0000?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/unixsock_posix.go:172 +0x16 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*UnixListener).Accept(0xc000fa8090) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/unixsock.go:260 +0x30 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc.(*Server).Serve(0xc000fa4000, {0x29f0d20, 0xc000fa8090}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: google.golang.org/grpc@v1.72.2/server.go:890 +0x469 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/server.ListenAndServePodResources({0x29f8e48, 0xc000964370}, {0xc000f92030, 0x30}, {{0x29f1f40, 0xc000f98000}, {0x7fbfb17894a0, 0xc000658608}, {0x7fbfb17894d0, 0xc000658608}, ...}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/server/server.go:255 +0x2bc Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).ListenAndServePodResources(0xc00036fc08, {0x29f8e48, 0xc000964370}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3236 +0x358 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/cmd/kubelet/app.startKubelet in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:1279 +0x353 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 418 [IO wait]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb8ba1000, 0x72) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc000672000?, 0x380016?, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Accept(0xc000672000) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:613 +0x28c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).accept(0xc000672000) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/fd_unix.go:161 +0x29 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*TCPListener).accept(0xc000e26000) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/tcpsock_posix.go:159 +0x1b Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net.(*TCPListener).Accept(0xc000e26000) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/tcpsock.go:380 +0x30 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Server).Serve(0xc000e0c000, {0x29f0cf0, 0xc000e26000}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/server.go:3463 +0x30c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Server).ListenAndServe(0xc000e0c000) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/server.go:3389 +0x72 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.ListenAndServe(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/server.go:3704 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app.run.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:915 +0x190 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x42d5fc?, 0x3c3cb60?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41af8f?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000e03f50, {0x29d9080, 0xc000e08000}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000e08000?}, 0x0?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000d5f560, 0x12a05f200, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/cmd/kubelet/app.run in goroutine 1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/cmd/kubelet/app/server.go:914 +0x2053 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 588 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d12008, 0xb5?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d12008) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 207 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.setRequestCancel.func4() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:402 +0x85 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by net/http.setRequestCancel in goroutine 551 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:401 +0x41a Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 208 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.awaitLegacyCancel({0x29f8e48?, 0xc000615590?}, 0xc000918840, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:744 +0x72 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by net/http.(*Transport).roundTrip in goroutine 551 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:650 +0x865 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 400 [chan receive]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/utils/oomparser.(*OomParser).StreamOoms(0xc00097e7d0, 0xc000aec9a0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/utils/oomparser/oomparser.go:121 +0xd1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/oom.(*realWatcher).Start in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/oom/oom_watcher_linux.go:77 +0xb1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 401 [chan receive]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/oom.(*realWatcher).Start.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/oom/oom_watcher_linux.go:83 +0xa5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/oom.(*realWatcher).Start in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/oom/oom_watcher_linux.go:79 +0x12c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 546 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000520f50, {0x29d9080, 0xc000aab4d0}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005b06c0?, {0x29d9080?, 0xc000aab4d0?}, 0xc0?, 0xc0002f43c0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000e814a0, 0xdf8475800, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Forever(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:80 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/server/stats.(*fsResourceAnalyzer).Start.func1 in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/server/stats/fs_resource_analyzer.go:70 +0x109 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 547 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/allocation.(*manager).Run.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/allocation/allocation_manager.go:217 +0xb5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/allocation.(*manager).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/allocation/allocation_manager.go:215 +0x74 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 548 [chan receive (nil chan)]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager.(*volumeManager).Run(0xc0008c3a40, {0x29f8dd8, 0x3c3a320}, {0x29e72c8, 0xc0002ad2c0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:316 +0x336 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1851 +0x551 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 549 [runnable]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: syscall.Syscall(0x0, 0xd, 0xc000b12200, 0x300) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: syscall/syscall_linux.go:74 +0x25 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: syscall.read(0xc001554e40?, {0xc000b12200?, 0x7fc00064cf01?, 0x41a545?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: syscall/zsyscall_linux_amd64.go:736 +0x38 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: syscall.Read(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: syscall/syscall_unix.go:183 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.ignoringEINTRIO(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:738 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc001554e40, {0xc000b12200, 0x300, 0x300}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:161 +0x2ae Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).read(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os/file_posix.go:30 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).Read(0xc0009ac8d8, {0xc000b12200?, 0x2037300?, 0xc000b12000?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os.readFileContents(0x0, 0xc000cdeac8) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:908 +0x90 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os.ReadFile({0x25e975d?, 0x41e6b1?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:871 +0xd3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/mount-utils.readMountInfo({0x25e975d, 0xc}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/mount-utils/mount_helper_unix.go:255 +0x50 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/mount-utils.ListProcMounts({0x25e975d?, 0x222ffc0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/mount-utils/mount_linux.go:727 +0x18 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/mount-utils.(*Mounter).List(0x7fc00064e210?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/mount-utils/mount_linux.go:427 +0x1f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm.validateSystemRequirements({0x2a0ac78?, 0xc000972460?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:161 +0x36 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm.(*containerManagerImpl).setupNode(0xc000658608, {0x29f8dd8, 0x3c3a320}, 0xc00174a4f0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:505 +0x75 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm.(*containerManagerImpl).Start(0xc000658608, {0x29f8dd8, 0x3c3a320}, 0xc001246a88, 0xc00174a4f0, 0xc00174a500, {0x29e72c8, 0xc0002ad2c0}, {0x7fbfb8104360, 0xc000482600}, ...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:664 +0x73c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initializeRuntimeDependentModules(0xc00036fc08, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1766 +0x283 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateRuntimeUp.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3169 +0x1f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Once).doSlow(0xc00041d810?, 0xc0006887e0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync/once.go:78 +0xac Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Once).Do(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync/once.go:69 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateRuntimeUp(0xc00036fc08) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3168 +0x625 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run.func2() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1866 +0x1c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1863 +0x5a9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 550 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc000ac3260}, 0xc000e00ee0, {0x29d9080, 0xc000dc0c00}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000dc0c00?}, 0xe0?, 0xaac89d?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000b01770, 0x5f5e100, 0x0, 0x1, 0xc000ac3260) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).fastStatusUpdateOnce(0xc00036fc08) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3259 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1870 +0x5eb Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 551 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc00093a3c0, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc0004bcd20?, 0xc00145b6c0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc00093a3c0, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc00093a3c0, {0x28?, 0x90?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc0006155e0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc00093a3c0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc000dc0900?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc000365d40, 0xc00093a280) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc00093a140, {0x29dc1a0, 0xc000365d40}, {0xc000909501?, 0x4adef3?, 0x3c16b20?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000744c00, 0xc00093a140, {0x2587380?, 0xc000909658?, 0x3c16b20?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000744c00, 0xc00093a140) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).request(0xc0005bc900, {0x29f8eb8, 0xc000354150}, 0xc000909a08) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1088 +0x3c5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Do(0xc0005bc900, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1136 +0x127 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Get(0x2a180c0, {0x29f8dd8, 0x3c3a320}, {0x7ffdae46ea6b, 0x2a}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, ...}}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:165 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease.(*controller).ensureLease(0xc0008c3ae0, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease/controller.go:215 +0x5c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease.(*controller).backoffEnsureLease(0xc0008c3ae0, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease/controller.go:196 +0x65 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease.(*controller).sync(0xc0008c3ae0, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease/controller.go:174 +0x1d7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f8dd8?, 0x3c3a320?}, 0xc000dc0630?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f8dd8, 0x3c3a320}, 0xc000909f98, {0x29d9080, 0xc000dc0630}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f8dd8, 0x3c3a320}, 0xc000089798, 0x2540be400, 0x3fa47ae147ae147b, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease.(*controller).Run(0xc0000897d0?, {0x29f8dd8?, 0x3c3a320?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/component-helpers/apimachinery/lease/controller.go:148 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1873 +0x677 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 552 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x29f8dd8, 0x3c3a320}, {0x29f1590, 0xc0005a48a0}, 0x1, 0x0, 0xc000e01f98) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/loop.go:66 +0x1d0 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x29f8dd8, 0x3c3a320}, 0x0?, 0x1, 0xc000116798) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:33 +0x56 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).fastStaticPodsRegistration(0xc00036fc08, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3378 +0x69 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1880 +0x6d1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 553 [sync.Mutex.Lock]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:95 +0x25 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/sync.(*Mutex).lockSlow(0xc000370540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/sync/mutex.go:149 +0x15d Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/sync.(*Mutex).Lock(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/sync/mutex.go:70 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Mutex).Lock(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync/mutex.go:46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateRuntimeUp(0xc00036fc08) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:3129 +0x5a Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x0?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41aed4?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000e02f50, {0x29d9080, 0xc000dc1080}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc000dc1080?}, 0x0?, 0x0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000a87100, 0x12a05f200, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run in goroutine 415 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet.go:1882 +0x77f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 534 [IO wait]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bb9800, 0x72) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc000881020?, 0xc000ee0000?, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc000881020, {0xc000ee0000, 0x2000, 0x2000}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).read(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os/file_posix.go:30 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os.(*File).Read(0xc00052e1b0, {0xc000ee0000?, 0x3b?, 0x7fc00064c108?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: os/file.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/euank/go-kmsg-parser/kmsgparser.(*parser).Parse.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/euank/go-kmsg-parser@v2.0.0+incompatible/kmsgparser/kmsgparser.go:129 +0xb1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/euank/go-kmsg-parser/kmsgparser.(*parser).Parse in goroutine 400 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/euank/go-kmsg-parser@v2.0.0+incompatible/kmsgparser/kmsgparser.go:123 +0x78 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 562 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x29f9198, 0x0}, 0xc000b005f8, 0xc000e06e40) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:210 +0xd1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.poll({0x29f9198, 0x0}, 0xd0?, 0xc000b005f8, 0xc000b00640) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x89 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x29f9198?, 0x0?}, 0xc000580008?, 0xc000580540?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x45 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0xffffffffffffffff?, 0xc00035b9e0?, 0xc000b00698?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x36 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.WaitForCacheSync(0xc0005c70a0?, {0xc000b00758?, 0xc000b006e0?, 0xc000882200?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:385 +0x49 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers.(*sharedInformerFactory).WaitForCacheSync(0xc0005c70a0?, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:193 +0xec Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume.(*VolumePluginMgr).Run(0x0?, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/volume/plugins.go:960 +0x76 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/volumemanager.(*volumeManager).Run in goroutine 548 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:305 +0x10e Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 563 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x29f8dd8, 0x3c3a320}, {0x29f1590, 0xc000ab6e60}, 0x0, 0x0, 0xc000c19f58) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/loop.go:66 +0x1d0 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x29f8dd8, 0x3c3a320}, 0xc0004c9eb0?, 0x0, 0xc000c19f58) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:33 +0x56 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/populator.(*desiredStateOfWorldPopulator).Run(0xc000d2d860, {0x29f8dd8, 0x3c3a320}, {0x29e72c8, 0xc0002ad2c0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:147 +0xc9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/volumemanager.(*volumeManager).Run in goroutine 548 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:308 +0x1e5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 564 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc000cba3c0, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc00092c280?, 0xc00145b6c0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc000cba3c0, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc000cba3c0, {0x78?, 0x10?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc000ab9d10?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc000cba3c0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc000cebad0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0004d1b00, 0xc000cba280) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc000cba280, {0x29dc1a0, 0xc0004d1b00}, {0x41ae01?, 0x7fbfb8157a00?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000666ab0, 0xc000cba280, {0x2587380?, 0xc0009e16a8?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000666ab0, 0xc000cba280) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).request(0xc0007d9200, {0x29f8dd8, 0x3c3a320}, 0xc0009e1a58) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1088 +0x3c5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Do(0xc0007d9200, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:1136 +0x127 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Get(0x2a17b80, {0x29f8dd8, 0x3c3a320}, {0x7ffdae46ea6b, 0x2a}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, ...}}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:165 +0x185 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).updateReconstructedFromNodeStatus(0xc00098aa00, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconstruct.go:185 +0xfc Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).reconcile(0xc00098aa00, {0x29f8dd8, 0x3c3a320}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:61 +0x105 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).Run.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:30 +0x1f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x261f0a5?, 0x1f?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0x0?}, 0x41aed4?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0x0}, 0xc0009e1ed0, {0x29d9080, 0xc000ceb830}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006d0500?, {0x29d9080?, 0xc000ceb830?}, 0x0?, 0xc0006d0500?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009e1f78, 0x5f5e100, 0x0, 0x1, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.(*reconciler).Run(0xc00098aa00, {0x29f8dd8, 0x3c3a320}, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:30 +0xdf Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/volumemanager.(*volumeManager).Run in goroutine 548 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:312 +0x2f5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 572 [sync.Cond.Wait]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_notifyListWait(0xc000f3a708, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:606 +0x159 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Cond).Wait(0x25f48e0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: sync/cond.go:71 +0x73 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*RealFIFO).PopBatch(0xc000f3a6e0, 0xc00097e040) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/the_real_fifo.go:279 +0x8b Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000f3a790, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:234 +0x162 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0x0?}, 0xc001074180?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0x0}, 0xc000655dc8, {0x29d9080, 0xc001074180}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f9198, 0x0}, 0xc000655dc8, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext(0xc000f3a790, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:199 +0x41e Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext(0xc00041dad0, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:579 +0x39a Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000a881e0?, 0xc000655fd0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:526 +0x1d Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:161 +0x56 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 562 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:159 +0x1df Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 582 [chan receive]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*processorListener).run(0xc0002a9220) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:1060 +0x4e Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 431 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 583 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002a9220) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:1029 +0x129 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 431 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 573 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext.poller.func1.1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x19b Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext.poller.func1 in goroutine 562 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xbb Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 431 [chan receive]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0002fbb80, {0x29f8e48, 0xc00105ce10}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:891 +0x65 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext.(*Group).StartWithContext.func4() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 572 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 432 [chan receive (nil chan)]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:167 +0x2c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/cache.(*controller).RunWithContext in goroutine 572 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:166 +0xdf Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 433 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc000d0c280, 0x0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc0004ee480?, 0xc00145b6c0?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc000d0c280, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc000d0c280, {0xc0?, 0xed?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc00105cf50?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc000d0c280) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc001074450?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0004d1b00, 0xc000d0c140) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc000d0c140, {0x29dc1a0, 0xc0004d1b00}, {0x41ae01?, 0x7fbfb16b3568?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000666ab0, 0xc000d0c140, {0x2587380?, 0xc0009df3f0?, 0x0?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000666ab0, 0xc000d0c140) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Watch(0xc000517980, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:794 +0x1a5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Watch(0xc000af47a8, {0x29f9198?, 0x0}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:201 +0x155 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/storage/v1.NewFilteredCSIDriverInformer.func4({0x29f9198, 0x0}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/storage/v1/csidriver.go:82 +0x12b Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*ListWatch).WatchWithContext(0xc000af4a00?, {0x29f9198?, 0x0?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/listwatch.go:309 +0x5d Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).watchList(0xc000d66000, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:771 +0x528 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).ListAndWatchWithContext(0xc000d66000, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:423 +0x1ff Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:369 +0x2e Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x42d5fc?, 0x3c3cb60?}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0x0?}, 0x41aed4?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0x0}, 0xc0009dfea8, {0x29d90a0, 0xc00105ce60}, 0x1) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2a02068?, {0x29d90a0?, 0xc00105ce60?}, 0x96?, 0x12?) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext(0xc000d66000, {0x29f9198, 0x0}) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:368 +0x22b Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 572 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 599 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe2b48, 0x65?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe2b48) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 793 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc00150c870) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 836 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 598 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe2908, 0xb5?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe2908) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 56 [chan receive]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/utils/oomparser.(*OomParser).StreamOoms(0xc000636800, 0xc00039c1c0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/utils/oomparser/oomparser.go:121 +0xd1 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*manager).watchForNewOoms in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:1205 +0xe7 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 57 [chan receive]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*manager).watchForNewOoms.func1() Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:1208 +0x58 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*manager).watchForNewOoms in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:1207 +0x135 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 58 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00048fd48, 0x1?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00048fd48) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 586 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f63f0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 538 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 587 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0008d0510) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 754 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 749 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc00023b680) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 183 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 183 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001416008, 0xb5?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001416008) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 559 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0009d7dd0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 598 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 184 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001416248, 0xbc?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001416248) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 185 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001416488, 0xbc?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001416488) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 186 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014166c8, 0x29?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014166c8) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 187 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001416908, 0x69?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001416908) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 188 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572008, 0xaf?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572008) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 189 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0008d21b0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 188 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 537 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe2d88, 0x28?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe2d88) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 59 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000920488, 0xb5?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000920488) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 577 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0009694d0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 59 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 538 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0002aa248, 0xb5?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0002aa248) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 754 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe2fc8, 0xb5?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe2fc8) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 755 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe3208, 0xbc?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe3208) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 600 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cc008, 0xaf?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cc008) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 756 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c61b0) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 600 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 601 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cc248, 0xd?, 0x5f5e100) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cc248) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: goroutine 602 [select]: Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b70630) Nov 01 20:03:08.984355 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 588 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 589 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d12248, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d12248) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 590 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d12488, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d12488) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 591 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d126c8, 0x1f?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d126c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 592 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d12908, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d12908) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 603 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b70870) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 592 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 593 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d12b48, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d12b48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 604 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014ccd88, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014ccd88) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 539 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c67e0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 604 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 193 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntilWithContext.poller.func1.1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x19b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.PollUntilWithContext.poller.func1 in goroutine 778 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xbb Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 61 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000921448, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000921448) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 605 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014ccfc8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014ccfc8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 606 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cd208, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cd208) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 607 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cd448, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cd448) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 60 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000921208, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000921208) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 757 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103eb48, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103eb48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 608 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cd688, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cd688) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 609 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cd8c8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cd8c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 786 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cdb08, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cdb08) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 787 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cdd48, 0x2a?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cdd48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 788 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001722008, 0x1c?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001722008) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 789 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001722248, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001722248) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 65 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2d208, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2d208) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 64 [chan receive]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:167 +0x2c Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/tools/cache.(*controller).RunWithContext in goroutine 780 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:166 +0xdf Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 790 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc00150c090) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 60 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 776 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x29f9198, 0xc0000c2540}, 0xc0009f1e20, 0xc001cb3e68) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:210 +0xd1 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.poll({0x29f9198, 0xc0000c2540}, 0x0?, 0xc0009f1e20, 0xc0009f1e68) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x89 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntilWithContext({0x29f9198?, 0xc0000c2540?}, 0x0?, 0x0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:111 +0x45 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntil(0x0?, 0x0?, 0x0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:98 +0x36 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/iptables.(*runner).Monitor(0xc00099a000, {0x25fac0d, 0x13}, {0xc001406f30, 0x3, 0x3}, 0xc00099e380, 0xdf8475800, 0xc0000c2540) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/iptables/iptables.go:604 +0x1bf Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initNetworkUtil in goroutine 415 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet_network_linux.go:55 +0x2d9 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 750 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015eb448, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015eb448) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 791 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0015ec750) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 750 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 63 [chan receive]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0002b9220, {0x29f8e48, 0xc0015010e0}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:891 +0x65 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext.(*Group).StartWithContext.func4() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 780 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 751 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015eb688, 0x23?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015eb688) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 752 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015eb8c8, 0x2a?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015eb8c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 753 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015ebb08, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015ebb08) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 778 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x29f9198, 0xc0000c2540}, 0xc00154fe20, 0xc001e02e68) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:210 +0xd1 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.poll({0x29f9198, 0xc0000c2540}, 0x0?, 0xc00154fe20, 0xc00154fe68) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x89 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntilWithContext({0x29f9198?, 0xc0000c2540?}, 0x0?, 0x0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:111 +0x45 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntil(0x0?, 0x0?, 0x0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:98 +0x36 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/iptables.(*runner).Monitor(0xc000f1a500, {0x25fac0d, 0x13}, {0xc0013d23c0, 0x3, 0x3}, 0xc00099e520, 0xdf8475800, 0xc0000c2540) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/util/iptables/iptables.go:604 +0x1bf Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initNetworkUtil in goroutine 415 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/kubelet_network_linux.go:55 +0x2d9 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 779 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/status.(*manager).Start.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/status/status_manager.go:257 +0xa6 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0x0?, 0x0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41aed4?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000af3f50, {0x29d9080, 0xc001797230}, 0x1) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x29d9080?, 0xc001797230?}, 0x0?, 0x0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000e26480, 0x0, 0x0, 0x1, 0xc0000c2540) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Forever(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:80 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/status.(*manager).Start in goroutine 415 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/status/status_manager.go:255 +0x15a Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 780 [sync.Cond.Wait]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: sync.runtime_notifyListWait(0xc000e34398, 0x0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: runtime/sema.go:606 +0x159 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: sync.(*Cond).Wait(0x25f48e0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: sync/cond.go:71 +0x73 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*RealFIFO).PopBatch(0xc000e34370, 0xc001542400) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/the_real_fifo.go:279 +0x8b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000e34420, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:234 +0x162 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0xc001848720?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000af6dc8, {0x29d9080, 0xc001848720}, 0x1) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc000af6dc8, 0x3b9aca00, 0x0, 0x1) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext(0xc000e34420, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/controller.go:199 +0x41e Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).RunWithContext(0xc00096c210, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:579 +0x39a Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0x75722f7368746170?, 0x6d65747379732d6e?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/shared_informer.go:526 +0x1d Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:161 +0x56 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 415 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/factory.go:159 +0x1df Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 781 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc00144bd50}, 0xc000da7f50, {0x29d9080, 0xc001797260}, 0x1) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x756b2f73656d756c?, {0x29d9080?, 0xc001797260?}, 0x65?, 0x612d6970612d6562?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0010b06f0, 0x3b9aca00, 0x0, 0x1, 0xc00144bd50) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:210 +0x7f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.Until(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:163 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/pleg.(*GenericPLEG).Start in goroutine 415 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/pleg/generic.go:161 +0x15b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 191 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.PollUntilWithContext.poller.func1.1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x19b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.PollUntilWithContext.poller.func1 in goroutine 776 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xbb Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 802 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).internalRoundTrip(0xc00145b6c0, 0xc001854640, 0x0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1030 +0x53f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).roundTrip(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:945 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc0009457a0?, 0xc00145b6c0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/clientconn.go:13 +0x15 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripViaPool(0xc000a8d900, 0xc001854640, {0x0?, 0x0?}, {0x29ebf50, 0xc000666a20}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:279 +0xf7 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).roundTripOpt(0xc000a8d900, 0xc001854640, {0xc0?, 0xad?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:413 +0xb2 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTripOpt(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:231 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Transport).RoundTrip(0xc000565380?, 0xc0018345f0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport_common.go:226 +0x17 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).roundTrip(0xc000565380, 0xc001854640) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:694 +0xa7f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).RoundTrip(0x23c3280?, 0xc0018489f0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/roundtrip.go:33 +0x18 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0004d1b00, 0xc001854500) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/transport/round_trippers.go:174 +0x2f7 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.send(0xc001854500, {0x29dc1a0, 0xc0004d1b00}, {0x41af01?, 0x7fbfb80c77e0?, 0x0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:259 +0x5e2 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).send(0xc000666ab0, 0xc001854500, {0x2587380?, 0xc0009eb3f0?, 0x0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:180 +0x91 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).do(0xc000666ab0, 0xc001854500) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:729 +0x9c9 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Client).Do(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/client.go:587 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest.(*Request).Watch(0xc000767e00, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/rest/request.go:794 +0x1a5 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype.(*Client[...]).Watch(0xc0018287a8, {0x29f9198?, 0xc0000c2540}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/gentype/type.go:201 +0x155 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/node/v1.NewFilteredRuntimeClassInformer.func4({0x29f9198, 0xc0000c2540}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/informers/node/v1/runtimeclass.go:82 +0x12b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*ListWatch).WatchWithContext(0x0?, {0x29f9198?, 0xc0000c2540?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/listwatch.go:309 +0x5d Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).watchList(0xc00028c0f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:771 +0x528 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).ListAndWatchWithContext(0xc00028c0f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:423 +0x1ff Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:369 +0x2e Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1({0xa?, 0xc000b7e7b0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x13 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1({0x29f9198?, 0xc0000c2540?}, 0x41af05?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:255 +0x51 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f9198, 0xc0000c2540}, 0xc0009ebea8, {0x29d90a0, 0xc001501130}, 0x1) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:256 +0xe5 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2a02068?, {0x29d90a0?, 0xc001501130?}, 0x96?, 0x12?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:233 +0x46 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*Reflector).RunWithContext(0xc00028c0f0, {0x29f9198, 0xc0000c2540}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache/reflector.go:368 +0x22b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:63 +0x1f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 780 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 803 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: context.(*cancelCtx).propagateCancel.func2() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: context/context.go:523 +0x98 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by context.(*cancelCtx).propagateCancel in goroutine 802 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: context/context.go:522 +0x409 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 804 [chan receive]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*persistConn).addTLS(0xc00182d7a0, {0x29f8e48, 0xc001834640}, {0xc0005df7a0, 0x4d}, 0xc00099bc80) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1712 +0x327 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).dialConn(0xc000565380, {0x29f8e48, 0xc001834640}, {{}, 0x0, {0xc0009457a0, 0x5}, {0xc0005df7a0, 0x52}, 0x0}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1793 +0x99c Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).dialConnFor(0xc000565380, 0xc0006fadc0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1618 +0xb8 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*Transport).startDialConnForLocked.func1() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1600 +0x35 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by net/http.(*Transport).startDialConnForLocked in goroutine 802 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1599 +0x112 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 881 [runnable]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb8ba0400, 0x72) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc00068a500?, 0xc000170b40?, 0x0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc00068a500, {0xc000170b40, 0x240, 0x240}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).Read(0xc00068a500, {0xc000170b40?, 0x0?, 0x240?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/fd_posix.go:68 +0x25 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net.(*conn).Read(0xc0009ac1c0, {0xc000170b40?, 0x41af05?, 0xc00188b760?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/net.go:196 +0x45 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*atLeastReader).Read(0xc000e0e180, {0xc000170b40?, 0xc00188b7d8?, 0x52395b?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:819 +0x3b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: bytes.(*Buffer).ReadFrom(0xc0018002a8, {0x29d7f00, 0xc000e0e180}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: bytes/buffer.go:217 +0x98 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readFromUntil(0xc001800008, {0x7fbfb8172818, 0xc000e0e0c0}, 0x0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:841 +0xde Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readRecordOrCCS(0xc001800008, 0x0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:630 +0x3db Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readRecord(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:592 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readHandshakeBytes(0xc001800008, 0x4) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1088 +0x51 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readHandshake(0xc001800008, {0x0, 0x0}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1099 +0x39 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).clientHandshake(0xc001800008, {0x29f8e48, 0xc001029720}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/handshake_client.go:332 +0x505 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).handshakeContext(0xc001800008, {0x29f8e48, 0xc001834640}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1580 +0x372 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).HandshakeContext(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1520 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http.(*persistConn).addTLS.func2() Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1706 +0x6e Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by net/http.(*persistConn).addTLS in goroutine 804 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/http/transport.go:1702 +0x309 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 810 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000646240) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 809 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 809 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000894008, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000894008) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 758 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000c88900) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 757 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 818 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2cb48, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2cb48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 819 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2cd88, 0xb8?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2cd88) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 759 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f106c0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 818 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 820 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2cfc8, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2cfc8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 836 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0009c98c8, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0009c98c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 783 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f10900) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 65 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 834 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d12d88, 0x1a?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d12d88) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 835 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d13448, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d13448) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 540 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001722908, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001722908) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 541 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6120) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 540 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 904 [IO wait]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.runtime_pollWait(0x7fbfb9bb9400, 0x72) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: runtime/netpoll.go:351 +0x85 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).wait(0xc000889b80?, 0xc000616a80?, 0x0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:84 +0x27 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*pollDesc).waitRead(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_poll_runtime.go:89 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll.(*FD).Read(0xc000889b80, {0xc000616a80, 0xa80, 0xa80}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: internal/poll/fd_unix.go:165 +0x279 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net.(*netFD).Read(0xc000889b80, {0xc000616a80?, 0x78614d0a20202020?, 0x6f6c20656c696620?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/fd_posix.go:68 +0x25 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net.(*conn).Read(0xc000592fe8, {0xc000616a80?, 0x41aed4?, 0x2078614d0a202020?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: net/net.go:196 +0x45 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*atLeastReader).Read(0xc00094e0d8, {0xc000616a80?, 0x2020202020303032?, 0x2020202020202020?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:819 +0x3b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: bytes.(*Buffer).ReadFrom(0xc000d317a8, {0x29d7f00, 0xc00094e0d8}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: bytes/buffer.go:217 +0x98 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readFromUntil(0xc000d31508, {0x7fbfb8172818, 0xc0005eff50}, 0xc0014fb9d0?) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:841 +0xde Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readRecordOrCCS(0xc000d31508, 0x0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:630 +0x3db Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).readRecord(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:592 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).Read(0xc000d31508, {0xc000c5b000, 0x1000, 0xc0014fbcf0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1397 +0x145 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: bufio.(*Reader).Read(0xc00196dec0, {0xc00028ec84, 0x9, 0x0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: bufio/bufio.go:245 +0x197 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: io.ReadAtLeast({0x29d8540, 0xc00196dec0}, {0xc00028ec84, 0x9, 0x9}, 0x9) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: io/io.go:335 +0x8e Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: io.ReadFull(...) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: io/io.go:354 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.readFrameHeader({0xc00028ec84, 0x9, 0x70?}, {0x29d8540?, 0xc00196dec0?}) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:250 +0x65 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc00028ec40) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:513 +0x6b Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc00028ec40) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/frame.go:572 +0x18 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0014fbfa8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1878 +0xca Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*ClientConn).readLoop(0xc00145b6c0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1760 +0x52 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 903 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:554 +0xda5 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 998 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00145c908, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00145c908) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 760 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0000d06c8, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0000d06c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 761 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f7440) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 760 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 837 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0009c9b08, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0009c9b08) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 838 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0009c9d48, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0009c9d48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 839 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cc488, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cc488) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 840 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cc6c8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cc6c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 544 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001723688, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001723688) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 545 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6750) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 544 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 821 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000921688, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000921688) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 866 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f10f30) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 821 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 822 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000921d48, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000921d48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 823 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118e008, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118e008) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 824 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118e248, 0x1c?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118e248) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 794 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010aa248, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010aa248) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 825 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000e22a20) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 794 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 795 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010aa488, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010aa488) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 796 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010aa6c8, 0x28?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010aa6c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 797 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010aa908, 0x26?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010aa908) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 798 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010aab48, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010aab48) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 826 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118ed88, 0xaf?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118ed88) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 827 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f113b0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 826 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 867 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001723b08, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001723b08) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 868 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6ab0) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 867 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 828 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118efc8, 0xb5?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118efc8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 869 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f11560) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 828 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 829 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118f208, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118f208) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 830 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118f448, 0x69?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118f448) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 831 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118f688, 0x22?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118f688) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 832 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118f8c8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118f8c8) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 833 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118fb08, 0x1d?, 0x5f5e100) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118fb08) Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: goroutine 882 [select]: Nov 01 20:03:09.007276 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118fd48, 0x69?, 0x5f5e100) Nov 01 20:03:09.005979 ip-10-0-138-177 systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118fd48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 883 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001514008, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001514008) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 884 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001514248, 0x69?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001514248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 885 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001514488, 0x24?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001514488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 886 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015146c8, 0xf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015146c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 887 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001514908, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001514908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 898 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001417d48, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001417d48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 899 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6e090) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 898 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 764 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe2248, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe2248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 841 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f7710) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 764 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 765 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe2488, 0x69?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe2488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 854 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0017886c8, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0017886c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 842 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc001069b00) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 854 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 855 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001788908, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001788908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 870 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0019846c8, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0019846c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 843 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6f30) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 870 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 871 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001984908, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001984908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 872 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c7170) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 871 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 900 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0019e2fc8, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0019e2fc8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 768 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6ea20) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 900 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 901 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0019e3208, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0019e3208) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 769 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe3448, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe3448) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 914 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f79e0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 769 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 813 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0012de488, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0012de488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 814 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000a17050) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 813 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 873 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0019e2248, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0019e2248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 915 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6e120) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 873 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 874 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0019e2488, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0019e2488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 888 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe26c8, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe26c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 889 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f62d0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 888 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 875 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0019e2b48, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0019e2b48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 876 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6e360) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 875 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 890 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe38c8, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe38c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 799 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f6ea0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 890 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 844 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001515448, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001515448) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 891 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f10630) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 844 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 845 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001515688, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001515688) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 892 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe3b08, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe3b08) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 800 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f70e0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 892 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 893 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000fe3d48, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000fe3d48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 801 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010ab688, 0xb1?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010ab688) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 846 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000e222d0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 801 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 930 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0010ab8c8, 0x69?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0010ab8c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 916 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0012de008, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0012de008) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 931 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6990) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 916 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 847 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001515b08, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001515b08) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 848 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f10cf0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 847 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 894 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001789208, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001789208) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 917 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0005f7680) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 894 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 895 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001789448, 0x20?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001789448) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 896 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001789688, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001789688) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 897 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0017898c8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0017898c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 946 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001789b08, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001789b08) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 947 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001789d48, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001789d48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 948 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00048efc8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00048efc8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 949 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014cc908, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014cc908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 950 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0014ccb48, 0x45?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0014ccb48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 951 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015ea008, 0x1f?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015ea008) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 932 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000d13d48, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000d13d48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 877 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000e22900) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 932 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 933 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000920908, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000920908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 934 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000920b48, 0xb8?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000920b48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 935 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000920d88, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000920d88) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 849 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2c908, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2c908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 902 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f110e0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 849 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 962 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2d448, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2d448) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 963 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2d688, 0x1d?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2d688) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 964 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2d8c8, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2d8c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 918 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0012defc8, 0xb1?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0012defc8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 965 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6fc0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 918 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 919 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0012df208, 0x29?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0012df208) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 920 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0012df448, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0012df448) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 966 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000f2db08, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000f2db08) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 967 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f11320) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 966 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 952 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00118e6c8, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00118e6c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 953 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000730750) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 952 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 968 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572248, 0xab?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 954 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f11680) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 968 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 969 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572488, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 970 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc0015726c8, 0x69?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc0015726c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 936 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc000921b08, 0xab?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc000921b08) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 971 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000e22cf0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 936 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 937 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001248008, 0x21?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001248008) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 938 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001248248, 0x69?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001248248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 939 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001248488, 0x69?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001248488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 878 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103f448, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103f448) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 940 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6ebd0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 878 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 879 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103fb08, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103fb08) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 880 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6ee10) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 879 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 972 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572b48, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572b48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 973 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f119e0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 972 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 978 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls.(*Conn).handshakeContext.func2() Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1556 +0x86 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by crypto/tls.(*Conn).handshakeContext in goroutine 881 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: crypto/tls/conn.go:1555 +0x265 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 856 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00170a248, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00170a248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 979 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b70000) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 856 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 857 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00170a488, 0x20?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00170a488) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 858 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00170a6c8, 0x33?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00170a6c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 859 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00170a908, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00170a908) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 921 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00170ab48, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00170ab48) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 860 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6e000) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 921 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 980 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103e008, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103e008) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 861 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6000) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 980 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 981 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103e248, 0xbc?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103e248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 982 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103e6c8, 0x1d?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103e6c8) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 862 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00145c008, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00145c008) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 863 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0009698c0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 862 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 983 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103ed88, 0xaf?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103ed88) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 984 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6360) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 983 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 864 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00145c248, 0xb5?, 0x5f5e100) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00145c248) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 865 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000969c20) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 864 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: goroutine 994 [select]: Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc0005bcc00, 0xc000cbb900, 0x0) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc0005bcc00, 0x1b1a725?, 0xc00145c248?) Nov 01 20:03:09.028322 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:09.006095 ip-10-0-138-177 systemd[1]: kubelet.service: Failed with result 'exit-code'. Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 376 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 985 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103efc8, 0xb5?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103efc8) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 922 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c65a0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 985 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 955 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc0007e4a80, 0xc000d0c280, 0x0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc0007e4a80, 0x1b1a725?, 0xc00170a908?) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 433 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 956 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc0007e5380, 0xc000cba3c0, 0x0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc0007e5380, 0x1b1a725?, 0xc0012df208?) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 564 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 957 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc000c84000, 0xc00049a3c0, 0x0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc000c84000, 0x1b1a725?, 0xc001789b08?) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 139 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 958 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc000c84180, 0xc000686a00, 0x0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc000c84180, 0x1b1a725?, 0xc001789b08?) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 166 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 959 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).writeRequest(0xc000c84300, 0xc000cbb540, 0x0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1199 +0xc85 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net/http2.(*clientStream).doRequest(0xc000c84300, 0x1b1a725?, 0xc001789b08?) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:1060 +0x18 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by golang.org/x/net/http2.(*ClientConn).internalRoundTrip in goroutine 404 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: golang.org/x/net@v0.56.0/http2/transport.go:966 +0x485 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 986 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103f688, 0xbc?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103f688) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 987 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103f8c8, 0x23?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103f8c8) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 988 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc00103fd48, 0x1a?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc00103fd48) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 989 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001722488, 0xbc?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001722488) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 990 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f102d0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 997 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 924 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000b6e480) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 923 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 907 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572908, 0xb5?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572908) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 925 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc00150c1b0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 907 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 908 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572d88, 0x22?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572d88) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 909 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001572fc8, 0x24?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001572fc8) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 910 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001573208, 0x69?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001573208) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 911 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001573448, 0xbc?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001573448) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 999 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000f10c60) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 998 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 974 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001248b48, 0xb5?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001248b48) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 991 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000e22360) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 974 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 975 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001248d88, 0xb5?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001248d88) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 992 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc000e225a0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 975 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 976 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001248fc8, 0x69?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001248fc8) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 993 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001722d88, 0xaf?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001722d88) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 977 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/common.(*realFsHandler).trackUsage(0xc0014c6900) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:135 +0x397 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/common.(*realFsHandler).Start in goroutine 993 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/common/fsHandler.go:144 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 1026 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001722fc8, 0x65?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001722fc8) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 1027 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeepingTick(0xc001723208, 0xbc?, 0x5f5e100) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:575 +0xbe Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*containerData).housekeeping(0xc001723208) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:533 +0x3f3 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*containerData).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/container.go:120 +0x4f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 816 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext({0x29f8dd8, 0x3c3a320}, 0xc0010b0e90, {0x29d9080, 0xc001c2d5c0}, 0x1) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:267 +0x185 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x29f8dd8, 0x3c3a320}, 0xc0010b0e90, 0xdf8475800, 0x0, 0x1) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:223 +0x8f Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/apimachinery/pkg/util/wait/backoff.go:172 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by k8s.io/kubernetes/pkg/kubelet/cm/dra.(*Manager).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: k8s.io/kubernetes/pkg/kubelet/cm/dra/manager.go:163 +0x17a Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 1218 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/container/raw.(*rawContainerWatcher).Start.func1() Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/raw/watcher.go:89 +0xd9 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/container/raw.(*rawContainerWatcher).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/container/raw/watcher.go:87 +0x269 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 926 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*manager).watchForNewContainers.func1() Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:1160 +0xbf Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*manager).watchForNewContainers in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:1158 +0x2b1 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 927 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*manager).globalHousekeeping(0xc00069c2c8, 0xc000d60230) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:403 +0xf4 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*manager).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:337 +0x5e5 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: goroutine 928 [select]: Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor/manager.(*manager).updateMachineInfo(0xc00069c2c8, 0xc000d603f0) Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:375 +0x95 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: created by github.com/google/cadvisor/manager.(*manager).Start in goroutine 549 Nov 01 20:03:09.047620 ip-10-0-138-177 kubenswrapper[5393]: github.com/google/cadvisor@v0.53.0/manager/manager.go:341 +0x6ed Nov 01 20:03:19.007168 ip-10-0-138-177 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1. Nov 01 20:03:19.007587 ip-10-0-138-177 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 20:03:19.017110 ip-10-0-138-177 systemd[1]: Starting Kubernetes Kubelet... Nov 01 20:03:19.083230 ip-10-0-138-177 kubenswrapper[5444]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:19.083230 ip-10-0-138-177 kubenswrapper[5444]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 20:03:19.083230 ip-10-0-138-177 kubenswrapper[5444]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:19.083230 ip-10-0-138-177 kubenswrapper[5444]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 20:03:19.085146 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085102 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.085146 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085146 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085150 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085154 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085157 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085161 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085164 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085167 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085170 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085173 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085176 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085178 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085181 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085184 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085187 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085191 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085194 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085197 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085200 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085203 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.085215 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085205 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085219 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085222 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085225 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085227 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085230 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085233 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085235 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085238 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085241 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085244 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085247 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085249 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085252 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085254 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085257 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085260 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085263 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085265 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.085723 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085268 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085271 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085273 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085276 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085278 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085281 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085283 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085286 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085289 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085292 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085295 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085297 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085300 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085302 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085305 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085308 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085311 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085313 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085316 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085319 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.086193 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085321 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085323 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085326 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085329 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085331 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085353 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085356 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085358 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085361 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085363 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085366 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085370 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085372 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085376 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085378 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085381 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085383 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085386 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085388 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.086702 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085391 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085393 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085396 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085398 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085401 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085404 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085406 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085409 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085412 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085415 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085417 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085420 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085422 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085425 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085427 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085430 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085432 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085435 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085437 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085441 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.087167 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085445 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085448 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085451 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085454 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085457 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085461 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085463 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.085467 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086195 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086202 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086205 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086209 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086211 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086214 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086217 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086219 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086222 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086224 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086227 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.087672 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086232 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086236 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086239 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086242 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086245 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086248 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086250 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086253 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086256 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086259 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086261 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086264 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086266 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086269 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086272 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086274 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086277 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086279 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086282 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086285 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.088163 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086287 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086290 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086293 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086296 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086299 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086302 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086304 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086307 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086310 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086313 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086315 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086318 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086320 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086323 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086325 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086328 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086331 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086333 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086336 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086339 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.088684 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086342 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086344 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086346 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086349 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086351 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086354 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086357 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086359 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086361 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086365 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086368 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086370 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086373 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086375 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086378 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086381 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086384 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086387 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086389 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.089207 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086392 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086395 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086398 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086400 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086403 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086405 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086408 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086411 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086413 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086416 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086418 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086421 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086423 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086426 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086430 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086434 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086437 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086440 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086443 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.089688 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086446 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086448 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086453 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086456 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086459 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086461 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086464 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086466 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086468 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086472 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086474 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086477 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086480 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086482 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086485 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086505 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086509 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086992 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.086998 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087002 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.090159 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087004 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087007 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087010 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087012 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087015 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087017 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087020 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087022 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087025 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087027 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087030 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087033 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087035 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087038 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087041 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087044 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087047 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087050 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087052 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087055 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.090690 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087058 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087061 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087063 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087066 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087069 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087071 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087073 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087076 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087078 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087081 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087084 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087087 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087089 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087092 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087094 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087097 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087100 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087103 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087105 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087108 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.091359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087111 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087114 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087116 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087119 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087122 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087124 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087127 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087129 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087132 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087134 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087137 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087139 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087142 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087144 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087147 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087150 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087153 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087156 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087158 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.092098 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087161 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087163 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087166 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087169 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087171 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087174 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087176 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087179 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087182 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087184 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087187 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087190 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087193 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087195 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087198 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087201 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087205 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087207 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087210 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.092580 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087214 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087216 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087220 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087224 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087227 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087230 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087233 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087235 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087237 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087240 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087243 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087246 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087249 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087251 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087254 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087257 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087259 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087262 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087264 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.093067 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087266 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087269 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087272 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087275 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087278 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.087280 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087365 5444 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087377 5444 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087383 5444 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087388 5444 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087392 5444 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087395 5444 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087400 5444 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087404 5444 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087407 5444 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087410 5444 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087413 5444 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087417 5444 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087421 5444 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087424 5444 flags.go:64] FLAG: --cgroup-root="" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087427 5444 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087430 5444 flags.go:64] FLAG: --client-ca-file="" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087433 5444 flags.go:64] FLAG: --cloud-provider="external" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087437 5444 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 20:03:19.093535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087441 5444 flags.go:64] FLAG: --cluster-domain="" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087444 5444 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087447 5444 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087450 5444 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087453 5444 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087459 5444 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087462 5444 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087465 5444 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087469 5444 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087472 5444 flags.go:64] FLAG: --contention-profiling="false" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087475 5444 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087478 5444 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087481 5444 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087485 5444 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087489 5444 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087492 5444 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087495 5444 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087498 5444 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087501 5444 flags.go:64] FLAG: --enable-server="true" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087504 5444 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087509 5444 flags.go:64] FLAG: --event-burst="100" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087512 5444 flags.go:64] FLAG: --event-qps="50" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087515 5444 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087518 5444 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 20:03:19.094147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087521 5444 flags.go:64] FLAG: --eviction-hard="" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087526 5444 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087530 5444 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087533 5444 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087536 5444 flags.go:64] FLAG: --eviction-soft="" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087539 5444 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087542 5444 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087545 5444 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087549 5444 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087552 5444 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087555 5444 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087558 5444 flags.go:64] FLAG: --feature-gates="" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087563 5444 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087567 5444 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087571 5444 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087574 5444 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087577 5444 flags.go:64] FLAG: --healthz-port="10248" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087580 5444 flags.go:64] FLAG: --help="false" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087583 5444 flags.go:64] FLAG: --hostname-override="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087586 5444 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087589 5444 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087592 5444 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087596 5444 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 20:03:19.094798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087600 5444 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087602 5444 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087605 5444 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087608 5444 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087611 5444 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087614 5444 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087617 5444 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087620 5444 flags.go:64] FLAG: --kube-reserved="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087623 5444 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087627 5444 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087630 5444 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087648 5444 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087653 5444 flags.go:64] FLAG: --lock-file="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087657 5444 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087662 5444 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087666 5444 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087672 5444 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087675 5444 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087679 5444 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087683 5444 flags.go:64] FLAG: --logging-format="text" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087686 5444 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087689 5444 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087693 5444 flags.go:64] FLAG: --manifest-url="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087696 5444 flags.go:64] FLAG: --manifest-url-header="" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087701 5444 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 20:03:19.095362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087704 5444 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087708 5444 flags.go:64] FLAG: --max-pods="110" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087711 5444 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087714 5444 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087717 5444 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087720 5444 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087723 5444 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087726 5444 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087729 5444 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087735 5444 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087739 5444 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087742 5444 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087745 5444 flags.go:64] FLAG: --pod-cidr="" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087748 5444 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087751 5444 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087754 5444 flags.go:64] FLAG: --pods-per-core="0" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087757 5444 flags.go:64] FLAG: --port="10250" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087760 5444 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087763 5444 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0ebfd44319ea487b5" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087767 5444 flags.go:64] FLAG: --qos-reserved="" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087770 5444 flags.go:64] FLAG: --read-only-port="10255" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087774 5444 flags.go:64] FLAG: --register-node="true" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087777 5444 flags.go:64] FLAG: --register-with-taints="" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087780 5444 flags.go:64] FLAG: --registry-burst="10" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087783 5444 flags.go:64] FLAG: --registry-qps="5" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087786 5444 flags.go:64] FLAG: --reserved-cpus="" Nov 01 20:03:19.096025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087789 5444 flags.go:64] FLAG: --reserved-memory="" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087794 5444 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087797 5444 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087802 5444 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087805 5444 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087808 5444 flags.go:64] FLAG: --runonce="false" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087811 5444 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087814 5444 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087817 5444 flags.go:64] FLAG: --seccomp-default="false" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087820 5444 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087823 5444 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087826 5444 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087829 5444 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087832 5444 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087835 5444 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087838 5444 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087842 5444 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087845 5444 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087848 5444 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087851 5444 flags.go:64] FLAG: --system-cgroups="" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087855 5444 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087860 5444 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087863 5444 flags.go:64] FLAG: --tls-cert-file="" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087866 5444 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087870 5444 flags.go:64] FLAG: --tls-min-version="" Nov 01 20:03:19.096699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087873 5444 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087876 5444 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087879 5444 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087882 5444 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087885 5444 flags.go:64] FLAG: --v="2" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087889 5444 flags.go:64] FLAG: --version="false" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087893 5444 flags.go:64] FLAG: --vmodule="" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087897 5444 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.087900 5444 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088027 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088033 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088037 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088041 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088045 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088048 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088053 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088056 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088059 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088061 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088064 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088067 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.097496 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088070 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088073 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088075 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088078 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088081 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088083 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088086 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088088 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088091 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088094 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088096 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088099 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088101 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088104 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088107 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088110 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088113 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088115 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088118 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088120 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.098280 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088123 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088128 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088131 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088133 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088136 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088138 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088143 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088145 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088148 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088151 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088154 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088156 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088159 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088161 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088164 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088167 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088169 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088172 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088175 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088177 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.098853 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088180 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088182 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088185 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088188 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088190 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088194 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088198 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088201 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088204 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088207 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088210 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088212 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088215 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088219 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088221 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088224 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088227 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088229 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088233 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.099399 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088235 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088238 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088241 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088243 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088246 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088248 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088251 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088253 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088256 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088259 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088261 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088264 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088267 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088269 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088272 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088274 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088277 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088279 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088282 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088284 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.099941 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088287 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088289 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088292 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088294 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088297 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088299 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088303 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088306 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088309 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088311 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088314 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088318 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088320 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088323 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088325 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.088351 5444 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:19.100533 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088393 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088398 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088401 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088404 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088406 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088409 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088412 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088414 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088417 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088419 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088422 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088425 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088427 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088430 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088432 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088435 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088438 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088440 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088443 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.101040 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088445 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088448 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088450 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088454 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088457 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088460 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088463 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088465 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088468 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088472 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088475 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088478 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088482 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088486 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088490 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088493 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088496 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088499 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088502 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.101526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088505 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088507 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088510 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088513 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088515 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088518 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088520 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088523 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088525 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088528 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088531 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088533 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088536 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088538 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088541 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088543 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088547 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088550 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088553 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088555 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.102017 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088558 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088561 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088565 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088567 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088570 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088572 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088575 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088578 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088581 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088584 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088587 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088589 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088592 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088594 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088597 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088599 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088602 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088604 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088607 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088609 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.102513 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088612 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088615 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088617 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088619 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088622 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088625 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088628 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088632 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088648 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088651 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088654 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088656 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088659 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088662 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088666 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088669 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088671 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088674 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088676 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.103032 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088679 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088682 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088684 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088687 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088690 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088692 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088696 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088698 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.088701 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.093106 5444 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.093122 5444 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.093139 5444 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.093143 5444 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093200 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093206 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093209 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093213 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093216 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093219 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.104026 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093221 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093224 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093226 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093229 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093231 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093234 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093237 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093240 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093242 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093245 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093247 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093250 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093253 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093256 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093258 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093261 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093263 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093266 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093269 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.104756 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093271 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093274 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093277 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093280 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093284 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093286 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093289 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093292 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093295 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093297 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093300 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093302 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093305 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093308 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093310 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093314 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093318 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093322 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093326 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.105238 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093329 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093332 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093336 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093338 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093341 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093344 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093347 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093349 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093352 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093354 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093357 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093359 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093362 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093364 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093368 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093371 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093373 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093376 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093378 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.105745 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093381 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093384 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093386 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093389 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093391 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093394 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093397 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093399 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093402 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093404 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093407 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093409 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093412 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093414 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093417 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093419 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093427 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093430 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093433 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093435 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.106216 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093438 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093440 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093443 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093445 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093448 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093450 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093452 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093455 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093458 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093460 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093463 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093466 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093468 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093471 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093474 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093476 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093479 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093481 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093484 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093486 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.106816 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093489 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093491 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093494 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.093520 5444 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093700 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093707 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093710 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093712 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093716 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093719 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093722 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093724 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093727 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093729 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093732 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093734 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.107535 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093737 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093739 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093742 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093745 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093747 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093750 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093752 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093755 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093758 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093760 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093763 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093766 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093769 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093771 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093774 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093777 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093779 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093782 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093784 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093787 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.108091 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093789 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093792 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093794 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093797 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093799 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093802 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093805 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093808 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093811 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093813 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093816 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093819 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093821 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093824 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093826 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093829 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093831 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093833 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093836 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093839 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.108914 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093841 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093844 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093846 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093849 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093851 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093854 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093856 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093859 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093861 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093865 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093869 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093872 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093875 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093877 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093880 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093883 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093885 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093888 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093891 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.109611 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093894 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093897 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093900 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093902 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093905 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093907 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093910 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093912 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093915 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093917 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093920 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093923 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093925 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093928 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093930 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093933 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093935 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093938 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093940 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093943 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.110218 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093945 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093948 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093951 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093953 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093956 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093958 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093961 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093963 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093966 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093968 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093971 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093973 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093976 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093979 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.093983 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.110734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.094007 5444 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094040 5444 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094048 5444 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094051 5444 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094054 5444 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094056 5444 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094059 5444 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094062 5444 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094064 5444 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094067 5444 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094070 5444 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094072 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094075 5444 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094078 5444 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094080 5444 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094083 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094086 5444 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094089 5444 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094091 5444 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094094 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 20:03:19.111147 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094097 5444 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094100 5444 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094102 5444 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094105 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094107 5444 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094110 5444 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094112 5444 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094115 5444 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094117 5444 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094120 5444 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094122 5444 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094125 5444 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094128 5444 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094131 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094135 5444 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094137 5444 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094140 5444 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094143 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094145 5444 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094148 5444 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 20:03:19.111857 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094150 5444 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094153 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094156 5444 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094159 5444 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094161 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094164 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094167 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094169 5444 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094172 5444 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094174 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094177 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094179 5444 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094182 5444 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094185 5444 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094188 5444 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094192 5444 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094196 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094198 5444 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094201 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 20:03:19.112487 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094204 5444 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094206 5444 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094209 5444 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094211 5444 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094214 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094217 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094219 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094223 5444 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094226 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094228 5444 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094231 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094233 5444 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094236 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094238 5444 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094241 5444 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094243 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094245 5444 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094248 5444 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094250 5444 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094253 5444 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 20:03:19.113359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094255 5444 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094258 5444 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094262 5444 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094264 5444 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094267 5444 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094269 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094272 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094274 5444 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094277 5444 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094280 5444 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094282 5444 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094285 5444 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094287 5444 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094290 5444 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094292 5444 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094295 5444 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094297 5444 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094300 5444 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094303 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 20:03:19.114317 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094305 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094309 5444 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094313 5444 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094316 5444 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094319 5444 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094322 5444 feature_gate.go:386] unrecognized feature gate: Example Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094325 5444 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094327 5444 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:19.094330 5444 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.094380 5444 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.095318 5444 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.095368 5444 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.095538 5444 server.go:1014] "Starting client certificate rotation" Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.095658 5444 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.095695 5444 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:56:57 +0000 UTC" deadline="2028-04-23 18:26:07.583940118 +0000 UTC" Nov 01 20:03:19.115335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.095730 5444 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12934h22m48.488212257s" Nov 01 20:03:19.116025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.096027 5444 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:19.116025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.097093 5444 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 20:03:19.116025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.098397 5444 log.go:25] "Validated CRI v1 runtime API" Nov 01 20:03:19.116025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.100082 5444 log.go:25] "Validated CRI v1 image API" Nov 01 20:03:19.116025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.100767 5444 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 20:03:19.116025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.102576 5444 fs.go:135] Filesystem UUIDs: map[498a6f32-2f0b-4c65-b99d-e9484dd74add:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cb440668-8f60-45b8-8567-fce71e3e278c:/dev/nvme0n1p3] Nov 01 20:03:19.116386 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.102592 5444 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~projected/kube-api-access-v8kg5:{mountpoint:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~projected/kube-api-access-v8kg5 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~projected/kube-api-access-hbnqc:{mountpoint:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~projected/kube-api-access-hbnqc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0effeb7a-2a88-48ad-9df6-4cd132b14052/volumes/kubernetes.io~projected/kube-api-access-rpb4w:{mountpoint:/var/lib/kubelet/pods/0effeb7a-2a88-48ad-9df6-4cd132b14052/volumes/kubernetes.io~projected/kube-api-access-rpb4w major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~projected/kube-api-access-dcr84:{mountpoint:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~projected/kube-api-access-dcr84 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~projected/kube-api-access-6wh5x:{mountpoint:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~projected/kube-api-access-6wh5x major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48faedf2-1476-4c1a-842d-e4a6e5ceeb82/volumes/kubernetes.io~projected/kube-api-access-lk569:{mountpoint:/var/lib/kubelet/pods/48faedf2-1476-4c1a-842d-e4a6e5ceeb82/volumes/kubernetes.io~projected/kube-api-access-lk569 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a46610c-94b6-4ec2-a351-e83bf7b95032/volumes/kubernetes.io~projected/kube-api-access-kjhhp:{mountpoint:/var/lib/kubelet/pods/4a46610c-94b6-4ec2-a351-e83bf7b95032/volumes/kubernetes.io~projected/kube-api-access-kjhhp major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~projected/kube-api-access-hkcg8:{mountpoint:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~projected/kube-api-access-hkcg8 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~projected/kube-api-access-ttkdq:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~projected/kube-api-access-ttkdq major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b696755-be38-4546-82e1-9b485bdef642/volumes/kubernetes.io~projected/kube-api-access-pfqjw:{mountpoint:/var/lib/kubelet/pods/8b696755-be38-4546-82e1-9b485bdef642/volumes/kubernetes.io~projected/kube-api-access-pfqjw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/902e2a0d-1e93-45e1-87cf-fd428a800f92/volumes/kubernetes.io~projected/kube-api-access-98sd5:{mountpoint:/var/lib/kubelet/pods/902e2a0d-1e93-45e1-87cf-fd428a800f92/volumes/kubernetes.io~projected/kube-api-access-98sd5 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~projected/kube-api-access-55rgc:{mountpoint:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~projected/kube-api-access-55rgc major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~projected/kube-api-access-g96z8:{mountpoint:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~projected/kube-api-access-g96z8 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~secret/metrics-certs major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~projected/kube-api-access-w466g:{mountpoint:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~projected/kube-api-access-w466g major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2d65d12-0f39-4bb4-9ced-cce27246077c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b2d65d12-0f39-4bb4-9ced-cce27246077c/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~projected/kube-api-access-ddjh4:{mountpoint:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~projected/kube-api-access-ddjh4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e/volumes/kubernetes.io~projected/kube-api-access-hcntf:{mountpoint:/var/lib/kubelet/pods/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e/volumes/kubernetes.io~projected/kube-api-access-hcntf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c55e75ab-72d7-43bb-b1ff-e385180a3c70/volumes/kubernetes.io~projected/kube-api-access-k6j2h:{mountpoint:/var/lib/kubelet/pods/c55e75ab-72d7-43bb-b1ff-e385180a3c70/volumes/kubernetes.io~projected/kube-api-access-k6j2h major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~projected/kube-api-access-j658n:{mountpoint:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~projected/kube-api-access-j658n major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~projected/kube-api-access-b4fj4:{mountpoint:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~projected/kube-api-access-b4fj4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffffa9d9-1fea-40cb-85c6-1a1edb49470d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ffffa9d9-1fea-40cb-85c6-1a1edb49470d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:316 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/340213e1dc0d4f8268675acd6b4f180ed15011b6fa62184e0cc112798f69feac/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/21eec1735bedeeeae9e2e48167aa3609272fab3be4d0ba4ddea15ba607f9fa0f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0f85770fa7d68c6df8855ebe6f8c33072b29c22cf4e105fc0ab7dabe0b269a4c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/c6e3a57a9de0a4f7af82eff150c3eceb918d56cfcdbbfd2ba744861f90002c07/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a00bb701652477cb81d71b7cde9ae587ad5a7f9877247c835530cc3017c66ac5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/29207a8730322c165d1301633365b3a63ceefe30f23c8f54711e8e5477afe819/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/73e52df931bc4bee8aaa45fa44c1e20f18fbe98a073a10937e11261e3ef587a3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/935171b91e31794616b2de940a68928cabffa40110e25348f21b75f9a425b410/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/10b663c11bf55ac1c435add09dd2d4105d8b09806b5108f0bda41b5d4e703461/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/c38547b967371bd4fdfe11b9fcabd07db4a99a6a653f0cd40a166bda60c79677/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/4f03a91750834f8479a9068f57e06e25ab94e74e9dac12a9f84b7b3679e1c54b/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3af7cda4a2cbab75e3ffc0ed3d182cb1dd02aa686561f07ca399a0a1f86d5db0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e8515f2522d2ff68416d2aef2cad8234c1c28d63b575d00e5a04fb515e0f7e78/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/6ab7baeb1b77e0a2232a68ee66293240133ffc0c683e143a77ec960b96a85d75/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f6e440f52aa5f7a88925ad38e81e7bbb96abf69daf9d523e9be2783d3425d614/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7f78ecc213d158237f491923fbce5294f4a21d5f0ef100ca7d78b1688dc3522a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2f77d536693b529692e8acdd8e46122825fcbf4381168b80a219e872526e80bf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c0b772a76f2509d11c97d30deff5a0ee7f56de8914e502e23b554f916549bfbd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/810c804c63d9397e42bc8e89ecb5e92cba37f0aab8adc111afb056c0e2e5d365/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/586675c09889f39e752add40274fd1059b88a2682aa8a6b84b059e3e0d92f6d4/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/927e7b4cbbdef6e19eee1855b77e5960b01de2269f5d8ffb843f173e03ad2ec6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7c17aed72f7892919188f7967cec646012cf5b032971274ce94ff5b6d091d076/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4458fa112aca8537fbf366c9327a245082fef12f2698526ef0c8557453ae7839/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/72fe0398a782fd94b42ae5f476d05ed577dba91670741c9f7f1305b84849af23/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/f4f8e4b869640272c2a60725d0a7bfdd8fc43109d4024af7f53c19476f97d237/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/c78122cd303ed306eed0c0365fd4ef3190db64f3fbcc2ca27dd6f40f07a9d6e5/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/99357c8f959847d086d4337a5307492d08e694a04aa03b6b636dd25d454179b5/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/649e82c666e5462934e10b3b6a544377e8f774330e848b93c3fd438d849f6acf/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/974dd28e495e16490c4929fba30f55ff731f45ec4b209a67f78220492a122c6a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/0917e48f25c34342c935f2e5dd442fabc761c75c49e98a54076a76a32abd37ee/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/19fc62de49e8af899544a4b100446da36e83441d867a8c4593511221719838d7/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/6df19c3a96948ac11416d9c85cb794d821fc09777e6d30ae87a7c694a14a9f43/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/c27a0e62a5d83dce6bbb5f85f64211ba25b8e53c823abc42d37e2d6ded87d98b/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/c20f8fa53ffe3426a2780618f6c24c27e2cb4e171ac6d016587146cad1937a00/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/eaeb1f29af3306d8d1781f5d9b804ef11efec35affe2d12bd10f3bc85f3fa4a2/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/b8de982c2bd98fb4a84ca6760a81d3ac20583e384913faacb18c28f212ac730f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/164728de19701cce87fc839a426c99e82fddea082bcb977dddd9f4348ca47664/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/f938a58e92d09e349a2c4f6a5a3fb74277ed9453efc84660281fb45426560b67/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/c5945b3d85619c450e24bf40acd99d73d16029cdfc89aef3bdd3bae2ddf8202f/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/61a266377a29ccc1f3e744e2ced0f9921b606e48bd4f5a759a4094cdb1cf817f/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/8ae23b8f950ee94eae623fe2ff1c5aed320f1d61559d14f019f1b8c92bf4fbb4/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/d050250b60e1f064dc9cd59f54e1850b8d9519108ec72d0527d82f17e3cd27aa/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/43ca22d9525e1c9a62ffb6be6ef13412d83baebf1099f598a1cb0b04d77b752e/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/3f8e4a46ca0f2538c138fba5f74a040e02667c0b0f8c508277d17ece80373d7d/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/1c27b99fe67cd124474e1043916ed82ed84d415269c22c4d0350f46be0f0d69a/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/760ea025b8b556b20ff177fdfd1b215b72267ff9cd07db3ad7e32c378299f557/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/1909b8879475f721b07eff0fb71ba1ff3026563bf1d4363330d4c19358061072/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/54f10124d946ff93e67290d964930325a900fa9892ca5d4a41f2b4358ebcbeac/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/7351cb1f8c8c61cb5771ab58d16461893bafc475efaa2422345862ad817079e6/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/240b985e4df4a7b68bd723606a43b278dfaa3c3814065b65c4ef83902540e884/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/b8c88799ddfe5b3f695347f7c2e08495265feb5d1cb1f4b53dda0ef60a412370/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/ad8e71b5165d42eed8b0bf663dbfcb9ec0a4c2967c8af3621a4f171799e4fddb/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/722b6f462a0f37fbd6fd3cd33e5aadf256c1958d5ffbadf2d65ae390e59c6756/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/86ed076c828b1f89668d5ccdf93290bf12f44664fa5a6c0a229bf2e6b2b8e77c/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/4578fea3e1535fef72d95378771795c4f6dad0cd670c1c315a8e4079fc104b0e/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/984a456f4984ecadc73559a7c97a7473c2dc66662f854d96b4f5a476245115f8/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/d06c56b33542d876dd626faf118007fd9a348e7444653b6b15daa8dfe6d663d8/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/fb8b2aac7cbdb13846e5dc4583c120bce7425b07c80dbd71f1e9ae4cb23da028/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/416ce637099bbedd387a310b68b575b80723fb0a0280f1255e9556235bc07c5a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/be302bed46d7b7992fb05971bf67aea8ce8af5a6ebd074f806178d508377c829/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3075de4b5700679794665cdd2be482b2169123f7a6296909579d4bf6649df454/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f8486446616bd1be0ef01127ca8429fdb6ee7feebac29240c29a2acdbdf89b34/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2967edafa138288d4381b1772ea818dbd5410981d1fdb7556eaa61c0d7dad23e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/d549bc2b1fe005dca59a9e46c45a26303d870e789a1dbfde1a0e33dc2730c9ef/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/0c1cbfb4ff66883e17361eba80ad3fa1353cfa80e5db01d270f981f819ddce09/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/fd8f4148588a622636f204ca811273dbb73f6414a43c7f0858a1c2b9bc7611a5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/557c141f51623940c6da3788bc8c520c1ebfce42065805c6e432b2056c7e7689/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.110376 5444 manager.go:217] Machine: {Timestamp:2026-11-01 20:03:19.109955633 +0000 UTC m=+0.054793493 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100494 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af26b6daa089661be53f69db44de2 SystemUUID:ec2af26b-6daa-0896-61be-53f69db44de2 BootID:e67060a4-6958-4af0-8c76-085d4325ee95 Filesystems:[{Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bb7195d0245c6728b56c6e37e5388dfd5a5f890f51c97bc0e7a9d9ad1b406fb/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~projected/kube-api-access-ttkdq DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec7b30b2f6cef2310ae49656eb21a7c5290a9e1a85640764a09e6349bc8a2fd8/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3557eb15859af83eba5a9a6bbd5d3eb54bcaec0d94dcb127e1ac5b1c1f0ade5c/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eb97b1eac531f8ded7ea322831306aac43c61d572658f77e7c446c786ca7390/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:247 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/908d2bd7363bae4f4725db51fe7f84521e602a19dfe8553bdecd07641a2ed665/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d/volumes/kubernetes.io~projected/kube-api-access-ddjh4 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~projected/kube-api-access-hkcg8 DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0effeb7a-2a88-48ad-9df6-4cd132b14052/volumes/kubernetes.io~projected/kube-api-access-rpb4w DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f5b3682b421350692eefa3152145b5b91f48cb6af32426c56cf0e352f58044a/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd72783145cd98ea4a011db725f72c95e60ecc31da3d8bbb1212e82c8a958b36/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32d44e0546c695177bcb5907c5e1a822a516409c163b8abc2136d6a8a46be54e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~projected/kube-api-access-w466g DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volumes/kubernetes.io~projected/kube-api-access-dcr84 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27bed001936bb6deb7a4b9f2571248ea151c53a0ecd7c991cb082b331e1d69de/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~projected/kube-api-access-55rgc DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31c6793e57ffafc4fed7f5bf17dcf7055d8c46ea91fe6067f92b456f6bdde144/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e/volumes/kubernetes.io~projected/kube-api-access-hcntf DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:261 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/775e49608e09321f100ef81ca2ebfb016a4623ddbe3847e3931b748d982587c2/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b007fa95-0178-4da4-a163-387a0528b3d1/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2709fa597dae7afacb33d1db02017b3533fc51287ec18f388388e3b12fc9b6e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d5d9b87693154acf8d1496374b8362abfb061fd071e5227f10204d4956f79ba/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fef475fc8920d59ce26a8b5e8a510411d8319a67351a337d07a89700e13efef7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29b90e96-6395-4931-8526-2e9dcbc72e3f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48faedf2-1476-4c1a-842d-e4a6e5ceeb82/volumes/kubernetes.io~projected/kube-api-access-lk569 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:317 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~projected/kube-api-access-6wh5x DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8da39585aa3e2e3686beda8616ed85711a8478e95a65a9e42091ad39cfe1ae/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/993de2c9c1272f4c5a5264cb35ee1fe87238c3feb06298fd475e9a3ddd21d2e7/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd6e1cad-750a-416c-9a32-81bf6a048e75/volumes/kubernetes.io~projected/kube-api-access-b4fj4 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3eb7e9a5-c890-47bf-994a-db739d7e2817/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:258 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0ec98fbdb961e35b455be7cc6ac5bf05cbc853d715bcb65cb64babc54648f1c/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffffa9d9-1fea-40cb-85c6-1a1edb49470d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:316 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a46610c-94b6-4ec2-a351-e83bf7b95032/volumes/kubernetes.io~projected/kube-api-access-kjhhp DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c55e75ab-72d7-43bb-b1ff-e385180a3c70/volumes/kubernetes.io~projected/kube-api-access-k6j2h DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b696755-be38-4546-82e1-9b485bdef642/volumes/kubernetes.io~projected/kube-api-access-pfqjw DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~projected/kube-api-access-v8kg5 DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cec48ca-4b08-41d1-bfad-2df7138f4b90/volumes/kubernetes.io~projected/kube-api-access-hbnqc DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af45e616-0778-4869-9066-7acad174e56a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:260 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78496de5-ccaa-448e-8683-52ea3c9e5de1/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~projected/kube-api-access-j658n DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2d65d12-0f39-4bb4-9ced-cce27246077c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78d12c00-e8a7-4ac4-af79-b618b391618e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/afc73791-d8d2-4a7e-84c2-eb514db3fee1/volumes/kubernetes.io~projected/kube-api-access-g96z8 DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/902e2a0d-1e93-45e1-87cf-fd428a800f92/volumes/kubernetes.io~projected/kube-api-access-98sd5 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12ab677e6a884f7ac3cf76c2f8ab4bc09c2c07622fa3db3faf42460bcbe808de/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1070ff71a35d1cd MacAddress:a6:c7:1c:18:08:8e Speed:10000 Mtu:8901} {Name:12ab677e6a884f7 MacAddress:16:16:7f:66:7f:b9 Speed:10000 Mtu:8901} {Name:31c6793e57ffafc MacAddress:a2:21:d0:22:9e:1d Speed:10000 Mtu:8901} {Name:993de2c9c1272f4 MacAddress:4e:91:57:5c:7e:b0 Speed:10000 Mtu:8901} {Name:9bb7195d0245c67 MacAddress:56:0f:0f:8b:11:36 Speed:10000 Mtu:8901} {Name:afc994dbaf50181 MacAddress:66:a5:48:d1:5b:5f Speed:10000 Mtu:8901} {Name:b2709fa597dae7a MacAddress:66:7f:4f:e9:44:dd Speed:10000 Mtu:8901} {Name:bd72783145cd98e MacAddress:3e:e4:f8:d4:26:33 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:39:0a:0d:7c:f1 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:41:11:ef:98:71 Speed:0 Mtu:8901} {Name:ec7b30b2f6cef23 MacAddress:86:59:0d:33:ac:3a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:39:0a:0d:7c:f1 Speed:0 Mtu:9001} {Name:f0ec98fbdb961e3 MacAddress:0e:75:c4:49:09:a5 Speed:10000 Mtu:8901} {Name:fb8da39585aa3e2 MacAddress:ba:08:13:a7:f1:c3 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:d2:6b:78:64:83:e1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:ff:cc:cc:64:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.110742 5444 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.110787 5444 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111047 5444 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111072 5444 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-177.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111190 5444 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111198 5444 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111207 5444 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111225 5444 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111243 5444 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111374 5444 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111464 5444 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111496 5444 kubelet.go:493] "Attempting to sync node with API server" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111505 5444 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111515 5444 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111523 5444 kubelet.go:405] "Adding apiserver pod source" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111531 5444 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.111999 5444 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.112009 5444 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.112436 5444 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.115838 5444 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.115850 5444 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.115958 5444 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116239 5444 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116284 5444 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116414 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116433 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116443 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116452 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116460 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116469 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116479 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116489 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116499 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116519 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116541 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116556 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116593 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 20:03:19.116772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.116601 5444 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 20:03:19.119927 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.119912 5444 server.go:1263] "Started kubelet" Nov 01 20:03:19.120016 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.119977 5444 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 20:03:19.120093 ip-10-0-138-177 systemd[1]: Started Kubernetes Kubelet. Nov 01 20:03:19.122684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.120011 5444 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 20:03:19.122684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.120746 5444 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 20:03:19.122684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.121582 5444 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 20:03:19.122684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.122595 5444 server.go:317] "Adding debug handlers to kubelet server" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125017 5444 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125030 5444 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125156 5444 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:56:58 +0000 UTC" deadline="2028-05-12 23:09:05.44075887 +0000 UTC" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125180 5444 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13395h5m46.315582462s" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125193 5444 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125202 5444 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.125209 5444 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.125302 5444 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127263 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127294 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127308 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127320 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127331 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a46610c-94b6-4ec2-a351-e83bf7b95032" volumeName="kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127343 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127354 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d65d12-0f39-4bb4-9ced-cce27246077c" volumeName="kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127366 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127381 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127392 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc73791-d8d2-4a7e-84c2-eb514db3fee1" volumeName="kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127403 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127415 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127438 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127451 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127466 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127497 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127509 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127521 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9f1d631-6c85-4c27-b21e-9ddc9c71db4e" volumeName="kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127533 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127545 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c55e75ab-72d7-43bb-b1ff-e385180a3c70" volumeName="kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127557 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127568 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127580 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127591 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6e1cad-750a-416c-9a32-81bf6a048e75" volumeName="kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127602 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffffa9d9-1fea-40cb-85c6-1a1edb49470d" volumeName="kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127615 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca" seLinuxMountContext="" Nov 01 20:03:19.127696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127626 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129091 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129122 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129133 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129142 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129150 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b696755-be38-4546-82e1-9b485bdef642" volumeName="kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129159 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c55e75ab-72d7-43bb-b1ff-e385180a3c70" volumeName="kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129173 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd6e1cad-750a-416c-9a32-81bf6a048e75" volumeName="kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4" seLinuxMountContext="" Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.127705 5444 factory.go:55] Registering systemd factory Nov 01 20:03:19.129222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129208 5444 factory.go:223] Registration of the systemd container factory successfully Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129186 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129265 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cec48ca-4b08-41d1-bfad-2df7138f4b90" volumeName="kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129281 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129294 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b696755-be38-4546-82e1-9b485bdef642" volumeName="kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129305 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129312 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cec48ca-4b08-41d1-bfad-2df7138f4b90" volumeName="kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129320 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129328 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129336 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9f1d631-6c85-4c27-b21e-9ddc9c71db4e" volumeName="kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129344 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6fa94a-74a8-46e6-a813-997f4f507da9" volumeName="kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129351 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129359 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129370 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b35a0e4a-9ed1-4c99-94ec-d22e03bc305d" volumeName="kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129380 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129388 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3eb7e9a5-c890-47bf-994a-db739d7e2817" volumeName="kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129395 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48faedf2-1476-4c1a-842d-e4a6e5ceeb82" volumeName="kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129403 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129411 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129419 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129427 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129435 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d65d12-0f39-4bb4-9ced-cce27246077c" volumeName="kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129439 5444 factory.go:153] Registering CRI-O factory Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129446 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48faedf2-1476-4c1a-842d-e4a6e5ceeb82" volumeName="kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129455 5444 factory.go:223] Registration of the crio container factory successfully Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129464 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78496de5-ccaa-448e-8683-52ea3c9e5de1" volumeName="kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129475 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129487 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129501 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78496de5-ccaa-448e-8683-52ea3c9e5de1" volumeName="kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129515 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129522 5444 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129527 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129540 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78d12c00-e8a7-4ac4-af79-b618b391618e" volumeName="kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129552 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="902e2a0d-1e93-45e1-87cf-fd428a800f92" volumeName="kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129564 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129576 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129587 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" volumeName="kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129599 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cec48ca-4b08-41d1-bfad-2df7138f4b90" volumeName="kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129612 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0effeb7a-2a88-48ad-9df6-4cd132b14052" volumeName="kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129623 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29b90e96-6395-4931-8526-2e9dcbc72e3f" volumeName="kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129543 5444 factory.go:103] Registering Raw factory Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129660 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af45e616-0778-4869-9066-7acad174e56a" volumeName="kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129674 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc73791-d8d2-4a7e-84c2-eb514db3fee1" volumeName="kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs" seLinuxMountContext="" Nov 01 20:03:19.129623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129680 5444 manager.go:1199] Started watching for new ooms in manager Nov 01 20:03:19.130990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129686 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b007fa95-0178-4da4-a163-387a0528b3d1" volumeName="kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume" seLinuxMountContext="" Nov 01 20:03:19.130990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129697 5444 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c55e75ab-72d7-43bb-b1ff-e385180a3c70" volumeName="kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h" seLinuxMountContext="" Nov 01 20:03:19.130990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129709 5444 reconstruct.go:97] "Volume reconstruction finished" Nov 01 20:03:19.130990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.129716 5444 reconciler.go:29] "Reconciler: start to sync state" Nov 01 20:03:19.130990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.130223 5444 manager.go:319] Starting recovery of all containers Nov 01 20:03:19.149830 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.149792 5444 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 20:03:19.151161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.151143 5444 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 20:03:19.151161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.151163 5444 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 20:03:19.151334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.151182 5444 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 20:03:19.151334 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.151220 5444 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 20:03:19.193145 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.193120 5444 generic.go:356] "Generic (PLEG): container finished" podID="af45e616-0778-4869-9066-7acad174e56a" containerID="9fbfb6015e1e27fc25f2aadbbf492117a9dc927ccd7d39e74e9d4ed288673e5e" exitCode=0 Nov 01 20:03:19.198346 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.198323 5444 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="df41f5056a9e85c1be7e4409891609efd74e2f9ae54c663a4b93d8fcd69ba89a" exitCode=0 Nov 01 20:03:19.198346 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.198344 5444 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="6f082ebaab1f2cf4c76d7cd410f5d32f6022d55e3cda43d1dcd265655d62e533" exitCode=0 Nov 01 20:03:19.198484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.198352 5444 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="e25c61da0b3b5f6f5f190fa15899ff3f1fd18c1c8f9646ad6c3e65c0e0249eae" exitCode=0 Nov 01 20:03:19.198484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.198360 5444 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="7af9afa9774b9bedd33ebca95bfc9c001e26c0a6d95dfca4838448ab61b98c1d" exitCode=0 Nov 01 20:03:19.198484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.198367 5444 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="c45affffaf27c13b174d2d0da7581ba4b76031c250f846d20795e25135652f56" exitCode=0 Nov 01 20:03:19.198484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.198374 5444 generic.go:356] "Generic (PLEG): container finished" podID="0effeb7a-2a88-48ad-9df6-4cd132b14052" containerID="3c1afb9ed6c59fb51a39547376e7102edb6ff7ee861a2cc366bcc3207ac7c892" exitCode=0 Nov 01 20:03:19.206096 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.206066 5444 generic.go:356] "Generic (PLEG): container finished" podID="ec2f737d93958347cc920d4e5916d818" containerID="bc6296c1649bd5108fb4124fd3093edae08faad6af36d5c471879182568bb22e" exitCode=0 Nov 01 20:03:19.224419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.224405 5444 manager.go:324] Recovery completed Nov 01 20:03:19.225275 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.225251 5444 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:03:19.243223 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.243205 5444 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:19.244138 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.244122 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:19.244197 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.244149 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:19.244197 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.244160 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:19.246816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.246803 5444 cpu_manager.go:226] "Starting" policy="none" Nov 01 20:03:19.246856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.246817 5444 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 20:03:19.246856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.246824 5444 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:19.246856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.246838 5444 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 20:03:19.246985 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.246969 5444 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 20:03:19.247017 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.246988 5444 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 20:03:19.247017 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247007 5444 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 20:03:19.247017 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247012 5444 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 20:03:19.247017 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247017 5444 policy_none.go:50] "Start" Nov 01 20:03:19.247126 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247026 5444 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 20:03:19.247126 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247037 5444 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 20:03:19.247178 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247173 5444 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 20:03:19.247207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247180 5444 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 20:03:19.247207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247186 5444 policy_none.go:44] "Start" Nov 01 20:03:19.247625 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247611 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:19.247693 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247649 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:19.247693 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.247661 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:19.251477 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.251460 5444 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 20:03:19.251556 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.251528 5444 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 20:03:19.254114 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254099 5444 manager.go:341] "Starting Device Plugin manager" Nov 01 20:03:19.254187 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.254121 5444 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 20:03:19.254187 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254130 5444 server.go:87] "Starting device plugin registration server" Nov 01 20:03:19.254385 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254373 5444 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 20:03:19.254451 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254384 5444 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 20:03:19.254501 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254470 5444 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 20:03:19.254593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254559 5444 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 20:03:19.254669 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254599 5444 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 20:03:19.254669 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254606 5444 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 20:03:19.254766 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.254684 5444 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T20:03:19.25459076Z","UUID":"77952c7e-d403-4ff1-bebd-4d6931ab6905","Handler":null,"Name":"","Endpoint":""} Nov 01 20:03:19.256076 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.256058 5444 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 20:03:19.256266 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.256094 5444 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-177.us-west-1.compute.internal\" not found" Nov 01 20:03:19.258204 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.257096 5444 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 20:03:19.258204 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.257145 5444 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 20:03:19.267929 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.267911 5444 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:19.317374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.317352 5444 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:19.317833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.317815 5444 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:19.317924 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.317912 5444 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 20:03:19.354982 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.354965 5444 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 20:03:19.355819 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.355801 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 20:03:19.355891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.355830 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 20:03:19.355891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.355845 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 20:03:19.355891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.355882 5444 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.415759 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.415742 5444 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 20:03:19.452301 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452267 5444 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal"] Nov 01 20:03:19.452864 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452851 5444 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.452919 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452888 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"6f167604710ff032b6c087eabbd79a27","Type":"ContainerStarted","Data":"39c94a5e7dafbc89305d7e780a7654f3fa9a2878b6bb328d1a3f92112222fbcc"} Nov 01 20:03:19.452959 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452926 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"6f167604710ff032b6c087eabbd79a27","Type":"ContainerStarted","Data":"23d564a7665dbe944fd25b616bf2ee255a0d7b944370851352a100b61c9730d5"} Nov 01 20:03:19.452959 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452938 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"ec2f737d93958347cc920d4e5916d818","Type":"ContainerStarted","Data":"853be52b889669b5e952965bababa6db527e6efd4593d8fe7883ed0285a773f9"} Nov 01 20:03:19.452959 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452949 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"ec2f737d93958347cc920d4e5916d818","Type":"ContainerDied","Data":"bc6296c1649bd5108fb4124fd3093edae08faad6af36d5c471879182568bb22e"} Nov 01 20:03:19.453078 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.452962 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" event={"ID":"ec2f737d93958347cc920d4e5916d818","Type":"ContainerStarted","Data":"1214894d1145e5f28d2431d9d92ed1c79a7d630d2ab5e38b72b80c5d676e3837"} Nov 01 20:03:19.453078 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.453067 5444 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.467391 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.467375 5444 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.467447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.467423 5444 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.467883 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.467865 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeNotReady" Nov 01 20:03:19.467952 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.467893 5444 setters.go:546] "Node became not ready" node="ip-10-0-138-177.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T20:03:19Z","lastTransitionTime":"2026-11-01T20:03:19Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 20:03:19.513373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.513302 5444 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:19.513373 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.513350 5444 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.513796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.513785 5444 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:19.513846 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:19.513811 5444 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.523854 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.523834 5444 nodeinfomanager.go:471] existing CSINode "ip-10-0-138-177.us-west-1.compute.internal" is owned by different node (oldNodeID="804d2f58-30a8-4e5e-82c4-1f82323c9251", newNodeID=""), cleaning up... Nov 01 20:03:19.620019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.619993 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f167604710ff032b6c087eabbd79a27-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"6f167604710ff032b6c087eabbd79a27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.620114 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.620021 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.620114 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.620040 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.720956 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.720929 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.720956 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.720947 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.721095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.721005 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.721095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.721025 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f167604710ff032b6c087eabbd79a27-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"6f167604710ff032b6c087eabbd79a27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.721095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.721051 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f167604710ff032b6c087eabbd79a27-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"6f167604710ff032b6c087eabbd79a27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:19.721095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:19.721085 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec2f737d93958347cc920d4e5916d818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" (UID: \"ec2f737d93958347cc920d4e5916d818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:20.111879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.111852 5444 apiserver.go:52] "Watching apiserver" Nov 01 20:03:20.209787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.209768 5444 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:20.209913 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.209870 5444 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:20.214884 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.214867 5444 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 20:03:20.215610 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.215587 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8","openshift-image-registry/node-ca-zw4q4","openshift-monitoring/node-exporter-ll458","openshift-multus/multus-additional-cni-plugins-7zvvr","openshift-monitoring/monitoring-plugin-77f4f76594-z6p67","openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j","openshift-insights/insights-runtime-extractor-hz4mt","openshift-console/console-78bdf5648-bx2k9","openshift-dns/dns-default-rvk4h","openshift-network-diagnostics/network-check-target-h5w6p","kube-system/konnectivity-agent-c5z7h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal","openshift-multus/multus-ldqdw","openshift-console/console-7c49869bf6-pj89j","openshift-dns/node-resolver-qc9m9","kube-system/global-pull-secret-syncer-tn5w8","openshift-cluster-node-tuning-operator/tuned-4njz9","openshift-console/console-7dcf7fbf49-rdv7w","openshift-ingress-canary/ingress-canary-nbcrt","openshift-monitoring/thanos-querier-5867784ff5-9s6fg","openshift-network-operator/iptables-alerter-8wmlv","openshift-ovn-kubernetes/ovnkube-node-9hk7d","kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-88xp6"] Nov 01 20:03:20.229563 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.229544 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.269127 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269102 5444 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:20.269318 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:20.269149 5444 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:20.269778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269753 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 20:03:20.269778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269779 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.269990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269935 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.269990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269952 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 20:03:20.269990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269971 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4d28g\"" Nov 01 20:03:20.269990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.269999 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.269990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270014 5444 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 20:03:20.270213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270019 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9nhp\"" Nov 01 20:03:20.270213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270005 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.270213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270041 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 20:03:20.270213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270094 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 20:03:20.270213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270093 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q2mtd\"" Nov 01 20:03:20.270213 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:20.270129 5444 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.us-west-1.compute.internal" Nov 01 20:03:20.270839 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.270827 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 20:03:20.271703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.271689 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 20:03:20.271778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.271755 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5296q\"" Nov 01 20:03:20.271841 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.271814 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.271963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.271950 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.272097 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.272086 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nd7lz\"" Nov 01 20:03:20.272444 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.272431 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hf5nj\"" Nov 01 20:03:20.272491 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.272455 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fbbjp\"" Nov 01 20:03:20.272545 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.272431 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 20:03:20.273011 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.272998 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.273069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273011 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 20:03:20.273069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273015 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 20:03:20.273183 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273069 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xqbdw\"" Nov 01 20:03:20.273183 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273079 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.273183 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273071 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 20:03:20.273183 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273163 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.273183 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273166 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.273367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273194 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.273367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273235 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.273367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.273252 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 20:03:20.307564 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.307546 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 20:03:20.308252 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308233 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.308341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308234 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wxh2n\"" Nov 01 20:03:20.308341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308273 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.308341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308278 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.308341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308315 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.308341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308337 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 20:03:20.308535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308348 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9494p\"" Nov 01 20:03:20.308535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308493 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvh2v\"" Nov 01 20:03:20.308535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308502 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 20:03:20.308535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308531 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxsv6\"" Nov 01 20:03:20.308711 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308609 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 20:03:20.308711 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.308688 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 20:03:20.311282 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.311235 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.311315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.311274 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.311376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.311365 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.311414 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.311398 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.311760 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.311745 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 20:03:20.312017 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312001 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-95tvs\"" Nov 01 20:03:20.312083 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312018 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 20:03:20.312083 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312005 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 20:03:20.312200 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312176 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 20:03:20.312200 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312179 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.312295 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312253 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.312344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312316 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 20:03:20.312481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312450 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7w45q\"" Nov 01 20:03:20.312481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312478 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 20:03:20.312597 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312494 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 20:03:20.312597 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312534 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-877r8\"" Nov 01 20:03:20.312597 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312574 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 20:03:20.312761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312678 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 20:03:20.312761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312701 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rxxgh\"" Nov 01 20:03:20.312761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312740 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 20:03:20.312859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.312765 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 20:03:20.318702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.318689 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nsfw\"" Nov 01 20:03:20.319076 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.319064 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 20:03:20.319565 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.319526 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 20:03:20.319626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.319527 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 20:03:20.319626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.319532 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2j5jb\"" Nov 01 20:03:20.319626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.319582 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bspnd5hul0il6\"" Nov 01 20:03:20.319626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.319601 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vlgpz\"" Nov 01 20:03:20.321678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.321662 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 20:03:20.322671 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.322651 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 20:03:20.324111 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324087 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.324194 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324133 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.324194 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324166 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-systemd\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.324299 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324196 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-host\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.324299 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324227 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.324299 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324250 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sd5\" (UniqueName: \"kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.324299 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324279 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.324487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324307 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.324487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324335 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.324487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324364 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-slash\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.324487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324398 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.324487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324428 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.324487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324459 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324488 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324516 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-k8s-cni-cncf-io\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324540 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-kubelet\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324569 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324600 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-bin\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324655 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpb4w\" (UniqueName: \"kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324685 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-kubelet-config\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324710 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w466g\" (UniqueName: \"kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324738 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j2h\" (UniqueName: \"kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.324776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324768 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324796 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-var-lib-kubelet\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324825 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324850 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-systemd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324879 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324909 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-kubernetes\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324943 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-ovn\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324973 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjh4\" (UniqueName: \"kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.324998 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-modprobe-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325026 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-cnibin\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325056 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325085 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-node-exporter-wtmp\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325118 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rgc\" (UniqueName: \"kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325147 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-system-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325169 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysconfig\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325199 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325231 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325260 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b696755-be38-4546-82e1-9b485bdef642-hosts-file\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325289 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325314 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325344 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325374 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325405 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325438 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-bin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325461 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-lib-modules\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325491 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fj4\" (UniqueName: \"kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325520 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-conf-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325548 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-conf\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325571 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-systemd-units\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325601 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325629 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-host-slash\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325674 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325705 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325730 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-sys\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325768 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325798 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk569\" (UniqueName: \"kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325826 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325855 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-crio-socket\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.325873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325884 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325911 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325941 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcntf\" (UniqueName: \"kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.325972 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326277 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-tmp\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326329 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-data-volume\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326470 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b696755-be38-4546-82e1-9b485bdef642-tmp-dir\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326481 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b2d65d12-0f39-4bb4-9ced-cce27246077c-konnectivity-ca\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326684 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326760 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b007fa95-0178-4da4-a163-387a0528b3d1-metrics-tls\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.326898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.326806 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-serviceca\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.327258 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327049 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-original-pull-secret\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.327399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327382 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-daemon-config\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.327513 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327490 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.327592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327572 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.327747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327729 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cnibin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.327782 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327768 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-log-socket\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.327820 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327793 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.327820 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327801 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcr84\" (UniqueName: \"kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.327912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327829 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.327912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327855 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.327912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327883 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-etc-kubernetes\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.328063 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327922 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-kubelet\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328063 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327952 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-system-cni-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.328063 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.327984 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqjw\" (UniqueName: \"kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.328063 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328035 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-multus-certs\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.328063 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328039 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.328314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328065 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.328314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328065 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-tuned\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.328314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328091 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.328314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328149 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd6e1cad-750a-416c-9a32-81bf6a048e75-cert\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:03:20.328314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328230 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.328568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328346 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-iptables-alerter-script\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.328568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328375 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.328568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328452 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.328568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328458 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b2d65d12-0f39-4bb4-9ced-cce27246077c-agent-certs\") pod \"konnectivity-agent-c5z7h\" (UID: \"b2d65d12-0f39-4bb4-9ced-cce27246077c\") " pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:03:20.328568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328481 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.328568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328529 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328584 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovn-node-metrics-cert\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328583 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78d12c00-e8a7-4ac4-af79-b618b391618e-tmp\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.328768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328659 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328745 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wh5x\" (UniqueName: \"kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.328932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328781 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.328932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328785 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-env-overrides\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328827 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-var-lib-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328865 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.328932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328902 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-netd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.328932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328920 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96z8\" (UniqueName: \"kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328937 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-node-log\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328966 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.328982 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-dbus\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329022 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329067 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329083 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0effeb7a-2a88-48ad-9df6-4cd132b14052-cni-binary-copy\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.329173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329163 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.329398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329182 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-sys\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.329398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329208 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.329398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329239 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.329398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329259 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b007fa95-0178-4da4-a163-387a0528b3d1-tmp-dir\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.329398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329284 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttkdq\" (UniqueName: \"kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.329398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329375 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:20.329554 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329401 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cni-binary-copy\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.329554 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329493 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.329610 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329590 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.329658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329613 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.329700 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329663 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc73791-d8d2-4a7e-84c2-eb514db3fee1-metrics-certs\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:20.329731 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329694 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-root\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.329811 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329790 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.329849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329833 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-netns\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.329849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329840 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.329904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329859 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-grpc-tls\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.329904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329867 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-multus\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.329904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329899 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.329986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.329926 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.330056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330043 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.330086 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330055 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-etc-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.330115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330097 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.330148 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330121 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.330180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330166 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.330229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330217 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.330259 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330251 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.330342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330324 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.330379 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330335 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.330417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330375 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-device-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.330417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330370 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-script-lib\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.330483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330417 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29b90e96-6395-4931-8526-2e9dcbc72e3f-ovnkube-config\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.330483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330468 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-host\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.330548 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330508 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.330548 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330528 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-hostroot\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.330652 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330559 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.330652 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330572 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-socket-dir-parent\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.330734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330696 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.330734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330702 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-run\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.330816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330743 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.330816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330784 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnqc\" (UniqueName: \"kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.330906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330835 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/af45e616-0778-4869-9066-7acad174e56a-node-exporter-textfile\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.330906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330863 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.330977 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.330906 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-netns\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.331028 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.331016 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.331028 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.331021 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:20.331676 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.331662 5444 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 20:03:20.370218 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370163 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 20:03:20.370297 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370227 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 20:03:20.370297 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370254 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 20:03:20.370392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370327 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 20:03:20.370442 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370418 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-99s87\"" Nov 01 20:03:20.370490 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370480 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 20:03:20.370538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370489 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:03:20.370538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370507 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 20:03:20.370600 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.370555 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 20:03:20.371559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.371544 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/af45e616-0778-4869-9066-7acad174e56a-node-exporter-tls\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.376849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.376835 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/78d12c00-e8a7-4ac4-af79-b618b391618e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.378026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.378012 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af45e616-0778-4869-9066-7acad174e56a-metrics-client-ca\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.378064 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.378043 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b007fa95-0178-4da4-a163-387a0528b3d1-config-volume\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.379151 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.379138 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78d12c00-e8a7-4ac4-af79-b618b391618e-metrics-client-ca\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.379190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.379135 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3eb7e9a5-c890-47bf-994a-db739d7e2817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.380293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.380275 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.380331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.380281 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3eb7e9a5-c890-47bf-994a-db739d7e2817-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.381179 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.381165 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjh4\" (UniqueName: \"kubernetes.io/projected/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-kube-api-access-ddjh4\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.381535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.381520 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpb4w\" (UniqueName: \"kubernetes.io/projected/0effeb7a-2a88-48ad-9df6-4cd132b14052-kube-api-access-rpb4w\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.382006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.381992 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w466g\" (UniqueName: \"kubernetes.io/projected/b007fa95-0178-4da4-a163-387a0528b3d1-kube-api-access-w466g\") pod \"dns-default-rvk4h\" (UID: \"b007fa95-0178-4da4-a163-387a0528b3d1\") " pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:20.382300 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.382282 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j2h\" (UniqueName: \"kubernetes.io/projected/c55e75ab-72d7-43bb-b1ff-e385180a3c70-kube-api-access-k6j2h\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.382397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.382384 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") pod \"console-7dcf7fbf49-rdv7w\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:20.384143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.384117 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk569\" (UniqueName: \"kubernetes.io/projected/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-kube-api-access-lk569\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.384772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.384746 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wh5x\" (UniqueName: \"kubernetes.io/projected/3eb7e9a5-c890-47bf-994a-db739d7e2817-kube-api-access-6wh5x\") pod \"openshift-state-metrics-7bd7489957-wfm9j\" (UID: \"3eb7e9a5-c890-47bf-994a-db739d7e2817\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wfm9j" Nov 01 20:03:20.384979 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.384961 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcntf\" (UniqueName: \"kubernetes.io/projected/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-kube-api-access-hcntf\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.385589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385312 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcr84\" (UniqueName: \"kubernetes.io/projected/29b90e96-6395-4931-8526-2e9dcbc72e3f-kube-api-access-dcr84\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.385589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385369 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rgc\" (UniqueName: \"kubernetes.io/projected/af45e616-0778-4869-9066-7acad174e56a-kube-api-access-55rgc\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.385589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385384 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sd5\" (UniqueName: \"kubernetes.io/projected/902e2a0d-1e93-45e1-87cf-fd428a800f92-kube-api-access-98sd5\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.385589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385371 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fj4\" (UniqueName: \"kubernetes.io/projected/fd6e1cad-750a-416c-9a32-81bf6a048e75-kube-api-access-b4fj4\") pod \"ingress-canary-nbcrt\" (UID: \"fd6e1cad-750a-416c-9a32-81bf6a048e75\") " pod="openshift-ingress-canary/ingress-canary-nbcrt" Nov 01 20:03:20.385589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385454 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnqc\" (UniqueName: \"kubernetes.io/projected/0cec48ca-4b08-41d1-bfad-2df7138f4b90-kube-api-access-hbnqc\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.385589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385556 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttkdq\" (UniqueName: \"kubernetes.io/projected/78d12c00-e8a7-4ac4-af79-b618b391618e-kube-api-access-ttkdq\") pod \"thanos-querier-5867784ff5-9s6fg\" (UID: \"78d12c00-e8a7-4ac4-af79-b618b391618e\") " pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:20.385829 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385695 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96z8\" (UniqueName: \"kubernetes.io/projected/afc73791-d8d2-4a7e-84c2-eb514db3fee1-kube-api-access-g96z8\") pod \"network-metrics-daemon-88xp6\" (UID: \"afc73791-d8d2-4a7e-84c2-eb514db3fee1\") " pod="openshift-multus/network-metrics-daemon-88xp6" Nov 01 20:03:20.385829 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385710 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqjw\" (UniqueName: \"kubernetes.io/projected/8b696755-be38-4546-82e1-9b485bdef642-kube-api-access-pfqjw\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.385891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.385840 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhhp\" (UniqueName: \"kubernetes.io/projected/4a46610c-94b6-4ec2-a351-e83bf7b95032-kube-api-access-kjhhp\") pod \"network-check-target-h5w6p\" (UID: \"4a46610c-94b6-4ec2-a351-e83bf7b95032\") " pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:20.431943 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.431920 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-node-log\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.432013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.431952 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.432013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.431971 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-dbus\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.432013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.431986 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.432013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432001 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-sys\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432017 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432039 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432051 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432022 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-node-log\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432074 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-dbus\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432085 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-sys\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432074 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432112 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-root\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432128 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432166 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-root\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.432186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432150 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-netns\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432203 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-netns\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432208 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-multus\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432222 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432226 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432235 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432270 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-multus\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432294 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-etc-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432310 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432325 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432334 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-etc-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432346 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432378 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432400 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-registration-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432404 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-device-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432428 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-host\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432485 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48faedf2-1476-4c1a-842d-e4a6e5ceeb82-host\") pod \"node-ca-zw4q4\" (UID: \"48faedf2-1476-4c1a-842d-e4a6e5ceeb82\") " pod="openshift-image-registry/node-ca-zw4q4" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432499 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432523 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-hostroot\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432550 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-device-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.432559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432560 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432551 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-socket-dir-parent\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432591 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-hostroot\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432594 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-socket-dir-parent\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432594 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-run\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432599 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432653 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-run\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432656 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-netns\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432686 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-netns\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432711 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432740 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432779 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-systemd\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432793 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-host\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432810 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867gk\" (UniqueName: \"kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432824 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-host\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432825 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-slash\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432834 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-systemd\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432845 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-slash\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432864 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432882 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-k8s-cni-cncf-io\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432897 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-kubelet\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432912 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-bin\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432933 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-k8s-cni-cncf-io\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432938 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-kubelet-config\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432965 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-kubelet\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432969 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432966 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432991 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-bin\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433000 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-var-lib-kubelet\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433021 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.432978 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ffffa9d9-1fea-40cb-85c6-1a1edb49470d-kubelet-config\") pod \"global-pull-secret-syncer-tn5w8\" (UID: \"ffffa9d9-1fea-40cb-85c6-1a1edb49470d\") " pod="kube-system/global-pull-secret-syncer-tn5w8" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433033 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-var-lib-kubelet\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433040 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-systemd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433068 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-systemd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433073 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433076 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-kubernetes\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433097 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-ovn\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433112 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-modprobe-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433113 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-kubernetes\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433126 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-cnibin\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433133 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-ovn\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433142 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-node-exporter-wtmp\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433169 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-cnibin\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433172 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/af45e616-0778-4869-9066-7acad174e56a-node-exporter-wtmp\") pod \"node-exporter-ll458\" (UID: \"af45e616-0778-4869-9066-7acad174e56a\") " pod="openshift-monitoring/node-exporter-ll458" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433187 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-system-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433203 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysconfig\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.433195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433187 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-modprobe-d\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433224 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433238 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-system-cni-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433250 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433266 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysconfig\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433275 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b696755-be38-4546-82e1-9b485bdef642-hosts-file\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433301 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433316 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b696755-be38-4546-82e1-9b485bdef642-hosts-file\") pod \"node-resolver-qc9m9\" (UID: \"8b696755-be38-4546-82e1-9b485bdef642\") " pod="openshift-dns/node-resolver-qc9m9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433326 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcg8\" (UniqueName: \"kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433361 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-bin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433386 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-var-lib-cni-bin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433406 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-lib-modules\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433428 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433432 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78496de5-ccaa-448e-8683-52ea3c9e5de1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433451 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-conf-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433488 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-lib-modules\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433490 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-conf\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433497 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433525 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-systemd-units\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433541 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-host-slash\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433543 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-multus-conf-dir\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433556 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-sys\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433569 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f1d631-6c85-4c27-b21e-9ddc9c71db4e-host-slash\") pod \"iptables-alerter-8wmlv\" (UID: \"b9f1d631-6c85-4c27-b21e-9ddc9c71db4e\") " pod="openshift-network-operator/iptables-alerter-8wmlv" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433578 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433594 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433605 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-sys\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433605 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cec48ca-4b08-41d1-bfad-2df7138f4b90-etc-sysctl-conf\") pod \"tuned-4njz9\" (UID: \"0cec48ca-4b08-41d1-bfad-2df7138f4b90\") " pod="openshift-cluster-node-tuning-operator/tuned-4njz9" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433609 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-crio-socket\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433576 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-systemd-units\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433649 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433668 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-sys-fs\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433680 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-run-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433687 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cnibin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433706 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-log-socket\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433712 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b35a0e4a-9ed1-4c99-94ec-d22e03bc305d-crio-socket\") pod \"insights-runtime-extractor-hz4mt\" (UID: \"b35a0e4a-9ed1-4c99-94ec-d22e03bc305d\") " pod="openshift-insights/insights-runtime-extractor-hz4mt" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433723 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433727 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-cnibin\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433742 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433754 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-log-socket\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433762 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-etc-kubernetes\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433792 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-etc-kubernetes\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433802 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-socket-dir\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433821 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433828 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-kubelet\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433852 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-system-cni-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433858 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-kubelet\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433868 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-multus-certs\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433889 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433892 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0effeb7a-2a88-48ad-9df6-4cd132b14052-system-cni-dir\") pod \"multus-additional-cni-plugins-7zvvr\" (UID: \"0effeb7a-2a88-48ad-9df6-4cd132b14052\") " pod="openshift-multus/multus-additional-cni-plugins-7zvvr" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433895 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c55e75ab-72d7-43bb-b1ff-e385180a3c70-host-run-multus-certs\") pod \"multus-ldqdw\" (UID: \"c55e75ab-72d7-43bb-b1ff-e385180a3c70\") " pod="openshift-multus/multus-ldqdw" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433906 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433917 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433920 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-var-lib-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433939 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/902e2a0d-1e93-45e1-87cf-fd428a800f92-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzjt8\" (UID: \"902e2a0d-1e93-45e1-87cf-fd428a800f92\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzjt8" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433942 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433957 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-netd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433977 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433943 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-var-lib-openvswitch\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.434576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.433996 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29b90e96-6395-4931-8526-2e9dcbc72e3f-host-cni-netd\") pod \"ovnkube-node-9hk7d\" (UID: \"29b90e96-6395-4931-8526-2e9dcbc72e3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:20.487398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.487378 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcg8\" (UniqueName: \"kubernetes.io/projected/78496de5-ccaa-448e-8683-52ea3c9e5de1-kube-api-access-hkcg8\") pod \"monitoring-plugin-77f4f76594-z6p67\" (UID: \"78496de5-ccaa-448e-8683-52ea3c9e5de1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:20.487652 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.487622 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") pod \"console-7c49869bf6-pj89j\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:20.534871 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.534849 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.534935 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.534888 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.534935 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.534905 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535007 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535062 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535033 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535062 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535051 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867gk\" (UniqueName: \"kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535141 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535073 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535141 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535115 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535262 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535242 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535262 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535248 5444 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 20:03:20.535730 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535701 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535823 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535773 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.535881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.535822 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.536048 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.536030 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.537440 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.537414 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.537505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.537495 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.590451 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.590426 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-867gk\" (UniqueName: \"kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk\") pod \"console-78bdf5648-bx2k9\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:20.840272 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:20.840241 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:21.260530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:21.260495 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78bdf5648-bx2k9"] Nov 01 20:03:21.266514 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:03:21.266490 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb949df27_3f20_4d11_82aa_9b12e4d1af3b.slice/crio-b346f11f961f22764f2199962c2479543333c61bac0074682c2f3a4aeb14ab18 WatchSource:0}: Error finding container b346f11f961f22764f2199962c2479543333c61bac0074682c2f3a4aeb14ab18: Status 404 returned error can't find the container with id b346f11f961f22764f2199962c2479543333c61bac0074682c2f3a4aeb14ab18 Nov 01 20:03:21.435475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:21.435442 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rvk4h" Nov 01 20:03:22.217631 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:22.217597 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdf5648-bx2k9" event={"ID":"b949df27-3f20-4d11-82aa-9b12e4d1af3b","Type":"ContainerStarted","Data":"0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f"} Nov 01 20:03:22.217861 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:22.217665 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdf5648-bx2k9" event={"ID":"b949df27-3f20-4d11-82aa-9b12e4d1af3b","Type":"ContainerStarted","Data":"b346f11f961f22764f2199962c2479543333c61bac0074682c2f3a4aeb14ab18"} Nov 01 20:03:22.331762 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:22.331697 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78bdf5648-bx2k9" podStartSLOduration=9.331680396 podStartE2EDuration="9.331680396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:22.331105473 +0000 UTC m=+3.275943341" watchObservedRunningTime="2026-11-01 20:03:22.331680396 +0000 UTC m=+3.276518263" Nov 01 20:03:22.583529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:22.583500 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:24.301214 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:24.301187 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:25.082027 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:25.081998 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5867784ff5-9s6fg" Nov 01 20:03:25.505148 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:25.505079 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z6p67" Nov 01 20:03:26.966655 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:26.966612 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:26.967016 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:26.966826 5444 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:03:26.992841 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:26.992819 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:26.997217 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:26.997198 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:28.355364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:28.355337 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c5z7h" Nov 01 20:03:28.787648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:28.787618 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:28.787807 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:28.787753 5444 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 20:03:28.791245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:28.791221 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:03:28.959403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:28.959373 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dcf7fbf49-rdv7w"] Nov 01 20:03:29.893249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:29.893221 5444 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.us-west-1.compute.internal" event="NodeReady" Nov 01 20:03:30.033675 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:30.033630 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:03:39.979788 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:39.979751 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bdf5648-bx2k9"] Nov 01 20:03:41.564962 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:41.564933 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h5w6p" Nov 01 20:03:45.246694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.246656 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c49869bf6-pj89j" podUID="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" containerName="console" containerID="cri-o://c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb" gracePeriod=15 Nov 01 20:03:45.486528 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.486508 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:45.590724 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590698 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.590827 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590737 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.590827 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590759 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.590908 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590884 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.591004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590923 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.591004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590952 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.591004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.590981 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.591158 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591070 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:45.591217 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591202 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") pod \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\" (UID: \"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e\") " Nov 01 20:03:45.591217 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591201 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:03:45.591342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591314 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:45.591342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591324 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:45.591463 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591451 5444 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-trusted-ca-bundle\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.591516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591472 5444 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.591516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591494 5444 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-tmp\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.591516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591506 5444 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-oauth-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.591634 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.591561 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:45.593030 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.593008 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:45.593153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.593135 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "kube-api-access-j658n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:45.593208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.593164 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config" pod "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" (UID: "c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:45.692429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.692405 5444 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.692429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.692426 5444 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-oauth-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.692589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.692436 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j658n\" (UniqueName: \"kubernetes.io/projected/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-kube-api-access-j658n\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:45.692589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:45.692446 5444 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e-console-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:46.294939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.294902 5444 generic.go:356] "Generic (PLEG): container finished" podID="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" containerID="c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb" exitCode=0 Nov 01 20:03:46.295448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.294966 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c49869bf6-pj89j" Nov 01 20:03:46.295448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.294981 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49869bf6-pj89j" event={"ID":"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e","Type":"ContainerDied","Data":"c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb"} Nov 01 20:03:46.295448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.295029 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c49869bf6-pj89j" event={"ID":"c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e","Type":"ContainerDied","Data":"1070ff71a35d1cd59f3068a509a720b50a3cb5e4a86902f12d626d99604d6923"} Nov 01 20:03:46.295448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.295049 5444 scope.go:122] "RemoveContainer" containerID="c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb" Nov 01 20:03:46.303624 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.303607 5444 scope.go:122] "RemoveContainer" containerID="c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb" Nov 01 20:03:46.303891 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:46.303872 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb\": container with ID starting with c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb not found: ID does not exist" containerID="c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb" Nov 01 20:03:46.303960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.303904 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb"} err="failed to get container status \"c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb\": rpc error: code = NotFound desc = could not find container \"c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb\": container with ID starting with c607dbfc605a62db70b4929def8a90fb91524b67644171857d076d74b9a66aeb not found: ID does not exist" Nov 01 20:03:46.456333 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.456306 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c49869bf6-pj89j"] Nov 01 20:03:46.505617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:46.505590 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c49869bf6-pj89j"] Nov 01 20:03:47.155975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:47.155944 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" path="/var/lib/kubelet/pods/c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e/volumes" Nov 01 20:03:49.796425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:49.796383 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hk7d" Nov 01 20:03:53.978395 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:53.978352 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7dcf7fbf49-rdv7w" podUID="0b6fa94a-74a8-46e6-a813-997f4f507da9" containerName="console" containerID="cri-o://347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d" gracePeriod=15 Nov 01 20:03:54.217935 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.217914 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:54.246055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246005 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246039 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246056 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246079 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246097 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246158 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246185 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") pod \"0b6fa94a-74a8-46e6-a813-997f4f507da9\" (UID: \"0b6fa94a-74a8-46e6-a813-997f4f507da9\") " Nov 01 20:03:54.246474 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246365 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:03:54.246680 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246632 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:54.246792 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246742 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:54.246792 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.246756 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:54.248104 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.248081 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "kube-api-access-v8kg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:54.248502 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.248484 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:54.248583 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.248562 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert" pod "0b6fa94a-74a8-46e6-a813-997f4f507da9" (UID: "0b6fa94a-74a8-46e6-a813-997f4f507da9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:03:54.322372 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.322350 5444 generic.go:356] "Generic (PLEG): container finished" podID="0b6fa94a-74a8-46e6-a813-997f4f507da9" containerID="347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d" exitCode=0 Nov 01 20:03:54.322471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.322387 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcf7fbf49-rdv7w" event={"ID":"0b6fa94a-74a8-46e6-a813-997f4f507da9","Type":"ContainerDied","Data":"347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d"} Nov 01 20:03:54.322471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.322439 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dcf7fbf49-rdv7w" event={"ID":"0b6fa94a-74a8-46e6-a813-997f4f507da9","Type":"ContainerDied","Data":"afc994dbaf501818ba8f857bb1c0e4524566c713ec2271a3a9eae3af882f0b47"} Nov 01 20:03:54.322471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.322459 5444 scope.go:122] "RemoveContainer" containerID="347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d" Nov 01 20:03:54.322471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.322396 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dcf7fbf49-rdv7w" Nov 01 20:03:54.329452 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.329432 5444 scope.go:122] "RemoveContainer" containerID="347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d" Nov 01 20:03:54.329747 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:03:54.329718 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d\": container with ID starting with 347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d not found: ID does not exist" containerID="347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d" Nov 01 20:03:54.329819 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.329758 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d"} err="failed to get container status \"347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d\": rpc error: code = NotFound desc = could not find container \"347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d\": container with ID starting with 347c61af73391a5ba2510eeac11b34e2309a65a76d63c95c90bfec28cca0df6d not found: ID does not exist" Nov 01 20:03:54.346885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346860 5444 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-oauth-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.346885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346880 5444 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6fa94a-74a8-46e6-a813-997f4f507da9-tmp\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.347007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346891 5444 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-oauth-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.347007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346900 5444 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.347007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346909 5444 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.347007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346917 5444 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b6fa94a-74a8-46e6-a813-997f4f507da9-console-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.347007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.346926 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kg5\" (UniqueName: \"kubernetes.io/projected/0b6fa94a-74a8-46e6-a813-997f4f507da9-kube-api-access-v8kg5\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:03:54.443462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.443441 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dcf7fbf49-rdv7w"] Nov 01 20:03:54.447250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:54.447234 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dcf7fbf49-rdv7w"] Nov 01 20:03:55.155729 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:03:55.155703 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6fa94a-74a8-46e6-a813-997f4f507da9" path="/var/lib/kubelet/pods/0b6fa94a-74a8-46e6-a813-997f4f507da9/volumes" Nov 01 20:04:00.033431 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:04:00.033396 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:04:05.002350 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.002309 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-78bdf5648-bx2k9" podUID="b949df27-3f20-4d11-82aa-9b12e4d1af3b" containerName="console" containerID="cri-o://0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f" gracePeriod=15 Nov 01 20:04:05.237258 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.237234 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:04:05.318681 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318647 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318690 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318711 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk\" (UniqueName: \"kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318728 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318748 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318780 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318800 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.318848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318836 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config\") pod \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\" (UID: \"b949df27-3f20-4d11-82aa-9b12e4d1af3b\") " Nov 01 20:04:05.319205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.318976 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:04:05.319205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.319123 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:04:05.319383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.319258 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:04:05.319480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.319444 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:04:05.319539 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.319472 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:04:05.321013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.320989 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:04:05.321105 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.321096 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:04:05.321206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.321180 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk" pod "b949df27-3f20-4d11-82aa-9b12e4d1af3b" (UID: "b949df27-3f20-4d11-82aa-9b12e4d1af3b"). InnerVolumeSpecName "kube-api-access-867gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:04:05.361117 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.361084 5444 generic.go:356] "Generic (PLEG): container finished" podID="b949df27-3f20-4d11-82aa-9b12e4d1af3b" containerID="0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f" exitCode=0 Nov 01 20:04:05.361269 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.361123 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78bdf5648-bx2k9" Nov 01 20:04:05.361269 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.361159 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdf5648-bx2k9" event={"ID":"b949df27-3f20-4d11-82aa-9b12e4d1af3b","Type":"ContainerDied","Data":"0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f"} Nov 01 20:04:05.361269 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.361221 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78bdf5648-bx2k9" event={"ID":"b949df27-3f20-4d11-82aa-9b12e4d1af3b","Type":"ContainerDied","Data":"b346f11f961f22764f2199962c2479543333c61bac0074682c2f3a4aeb14ab18"} Nov 01 20:04:05.361269 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.361247 5444 scope.go:122] "RemoveContainer" containerID="0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f" Nov 01 20:04:05.369591 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.369516 5444 scope.go:122] "RemoveContainer" containerID="0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f" Nov 01 20:04:05.369930 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:04:05.369908 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f\": container with ID starting with 0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f not found: ID does not exist" containerID="0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f" Nov 01 20:04:05.370015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.369934 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f"} err="failed to get container status \"0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f\": rpc error: code = NotFound desc = could not find container \"0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f\": container with ID starting with 0785f8f8697fc97758813c585bde14d098dc291bb7f877d74e80e0cff4539f9f not found: ID does not exist" Nov 01 20:04:05.420046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420022 5444 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-oauth-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420043 5444 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-trusted-ca-bundle\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420054 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867gk\" (UniqueName: \"kubernetes.io/projected/b949df27-3f20-4d11-82aa-9b12e4d1af3b-kube-api-access-867gk\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420064 5444 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b949df27-3f20-4d11-82aa-9b12e4d1af3b-tmp\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420073 5444 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420082 5444 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420090 5444 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b949df27-3f20-4d11-82aa-9b12e4d1af3b-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.420201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.420099 5444 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b949df27-3f20-4d11-82aa-9b12e4d1af3b-console-oauth-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:04:05.526881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.526836 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78bdf5648-bx2k9"] Nov 01 20:04:05.528465 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:05.528443 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78bdf5648-bx2k9"] Nov 01 20:04:07.158108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:04:07.158071 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b949df27-3f20-4d11-82aa-9b12e4d1af3b" path="/var/lib/kubelet/pods/b949df27-3f20-4d11-82aa-9b12e4d1af3b/volumes" Nov 01 20:04:30.033332 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:04:30.033294 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:05:00.034570 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:05:00.034498 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:05:30.034107 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:05:30.034064 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:05:55.657383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657344 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw"] Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657685 5444 cpu_manager.go:414] "containerMap: removing container" podUID="b949df27-3f20-4d11-82aa-9b12e4d1af3b" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657699 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b949df27-3f20-4d11-82aa-9b12e4d1af3b" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657707 5444 cpu_manager.go:414] "containerMap: removing container" podUID="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657712 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657718 5444 cpu_manager.go:414] "containerMap: removing container" podUID="0b6fa94a-74a8-46e6-a813-997f4f507da9" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657723 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6fa94a-74a8-46e6-a813-997f4f507da9" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657777 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="b949df27-3f20-4d11-82aa-9b12e4d1af3b" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657785 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a53dce-f4f6-42cc-8fe6-52b35c8bb09e" containerName="console" Nov 01 20:05:55.657934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.657791 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6fa94a-74a8-46e6-a813-997f4f507da9" containerName="console" Nov 01 20:05:55.659227 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.659211 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.719188 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.719167 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 20:05:55.719341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.719324 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 20:05:55.722869 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.722854 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.722921 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.722878 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.722921 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.722913 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.723023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.723007 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.723064 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.723049 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.723098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.723071 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.723098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.723090 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvs79\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.767919 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.767898 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 20:05:55.767989 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.767951 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 20:05:55.767989 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.767956 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 20:05:55.768052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.767987 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 20:05:55.768166 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.768152 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q7njp\"" Nov 01 20:05:55.775116 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.775095 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw"] Nov 01 20:05:55.824280 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824251 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.824385 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824288 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.824385 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824310 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.824505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824434 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvs79\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.824505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824496 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.824613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824529 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.824613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.824594 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.825257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.825231 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.826654 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.826605 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.826758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.826613 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.826758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.826748 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.826872 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.826775 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.881088 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.881063 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvs79\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.881272 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.881255 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8wtjw\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:55.969772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:55.969716 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:05:56.388561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:56.388533 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw"] Nov 01 20:05:56.392305 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:05:56.392277 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c2061f_2923_4a25_81ff_761a97ed57f4.slice/crio-bd7fe8e2fc8526e43bebea5031be328573c6811b954b480a6f2e64655742d9cf WatchSource:0}: Error finding container bd7fe8e2fc8526e43bebea5031be328573c6811b954b480a6f2e64655742d9cf: Status 404 returned error can't find the container with id bd7fe8e2fc8526e43bebea5031be328573c6811b954b480a6f2e64655742d9cf Nov 01 20:05:56.394138 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:56.394121 5444 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:05:56.711238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:05:56.711173 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" event={"ID":"e6c2061f-2923-4a25-81ff-761a97ed57f4","Type":"ContainerStarted","Data":"bd7fe8e2fc8526e43bebea5031be328573c6811b954b480a6f2e64655742d9cf"} Nov 01 20:06:00.034203 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:06:00.034170 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:06:04.862516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:04.862486 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:04.862824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:04.862547 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:05.742793 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:05.742759 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" event={"ID":"e6c2061f-2923-4a25-81ff-761a97ed57f4","Type":"ContainerStarted","Data":"c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372"} Nov 01 20:06:05.742985 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:05.742923 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:06:05.744500 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:05.744463 5444 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8wtjw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:06:05.744622 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:05.744525 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" podUID="e6c2061f-2923-4a25-81ff-761a97ed57f4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:06:05.949932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:05.949884 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" podStartSLOduration=2.482505645 podStartE2EDuration="10.949872222s" totalImagesPullingTime="8.467366577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:05.909938336 +0000 UTC m=+166.854776218" watchObservedRunningTime="2026-11-01 20:06:05.949872222 +0000 UTC m=+166.894710084" Nov 01 20:06:06.746415 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:06.746384 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:06:30.033712 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:06:30.033602 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:06:38.064168 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.064128 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq"] Nov 01 20:06:38.071842 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.071824 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.112367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.112341 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.112463 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.112375 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.112463 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.112392 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52290942-7dee-4686-be29-66a41d5284c1-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.112538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.112485 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.112538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.112522 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxh7\" (UniqueName: \"kubernetes.io/projected/52290942-7dee-4686-be29-66a41d5284c1-kube-api-access-2mxh7\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.112611 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.112595 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52290942-7dee-4686-be29-66a41d5284c1-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.177851 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.177825 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 20:06:38.177851 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.177841 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 20:06:38.177851 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.177836 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 20:06:38.178200 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.178161 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 20:06:38.178200 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.178178 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 20:06:38.178477 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.178458 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sx9cg\"" Nov 01 20:06:38.180147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.180109 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq"] Nov 01 20:06:38.213165 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213144 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52290942-7dee-4686-be29-66a41d5284c1-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.213259 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213185 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.213259 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213203 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.213259 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213219 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52290942-7dee-4686-be29-66a41d5284c1-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.213259 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213239 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.213466 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213260 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxh7\" (UniqueName: \"kubernetes.io/projected/52290942-7dee-4686-be29-66a41d5284c1-kube-api-access-2mxh7\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.213594 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.213572 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52290942-7dee-4686-be29-66a41d5284c1-manifests\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.215352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.215331 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52290942-7dee-4686-be29-66a41d5284c1-tmp\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.215738 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.215720 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.216010 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.215992 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.216126 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.216107 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52290942-7dee-4686-be29-66a41d5284c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.276037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.276016 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxh7\" (UniqueName: \"kubernetes.io/projected/52290942-7dee-4686-be29-66a41d5284c1-kube-api-access-2mxh7\") pod \"opendatahub-operator-controller-manager-5d77b8c7cf-mshvq\" (UID: \"52290942-7dee-4686-be29-66a41d5284c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.386304 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.386255 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:38.894537 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:38.894508 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq"] Nov 01 20:06:38.898002 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:06:38.897978 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52290942_7dee_4686_be29_66a41d5284c1.slice/crio-31adb0a5cd26f28d90be59570cfde2155c68a8ec7fe779b145d46c52442cfac8 WatchSource:0}: Error finding container 31adb0a5cd26f28d90be59570cfde2155c68a8ec7fe779b145d46c52442cfac8: Status 404 returned error can't find the container with id 31adb0a5cd26f28d90be59570cfde2155c68a8ec7fe779b145d46c52442cfac8 Nov 01 20:06:39.851392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:39.851354 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" event={"ID":"52290942-7dee-4686-be29-66a41d5284c1","Type":"ContainerStarted","Data":"31adb0a5cd26f28d90be59570cfde2155c68a8ec7fe779b145d46c52442cfac8"} Nov 01 20:06:42.863243 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:42.863217 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" event={"ID":"52290942-7dee-4686-be29-66a41d5284c1","Type":"ContainerStarted","Data":"4550ea9d3934363a50b1f79eb7397153ba7d98fbd4927e8255ec2d46442d64bf"} Nov 01 20:06:43.868169 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:43.868128 5444 generic.go:356] "Generic (PLEG): container finished" podID="52290942-7dee-4686-be29-66a41d5284c1" containerID="4550ea9d3934363a50b1f79eb7397153ba7d98fbd4927e8255ec2d46442d64bf" exitCode=0 Nov 01 20:06:43.868653 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:43.868181 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" event={"ID":"52290942-7dee-4686-be29-66a41d5284c1","Type":"ContainerDied","Data":"4550ea9d3934363a50b1f79eb7397153ba7d98fbd4927e8255ec2d46442d64bf"} Nov 01 20:06:44.873608 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:44.873576 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" event={"ID":"52290942-7dee-4686-be29-66a41d5284c1","Type":"ContainerStarted","Data":"ed4fa3ef08f6687f92c4e8948ee7384394f0ac448ef7fa716e8c4093bef6c507"} Nov 01 20:06:44.873976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:44.873699 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:06:45.039915 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:45.039867 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" podStartSLOduration=2.935574156 podStartE2EDuration="8.039851921s" totalImagesPullingTime="4.683798238s" totalInitContainerRuntime="420.479527ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:45.038380022 +0000 UTC m=+205.983217890" watchObservedRunningTime="2026-11-01 20:06:45.039851921 +0000 UTC m=+205.984689787" Nov 01 20:06:53.733001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.732961 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-st85q"] Nov 01 20:06:53.737323 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.737301 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:53.821181 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.821155 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck5bx\" (UniqueName: \"kubernetes.io/projected/bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96-kube-api-access-ck5bx\") pod \"jobset-operator-674cd59dc5-st85q\" (UID: \"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:53.821312 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.821199 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96-tmp\") pod \"jobset-operator-674cd59dc5-st85q\" (UID: \"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:53.842049 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.842028 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:06:53.842128 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.842078 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-v755l\"" Nov 01 20:06:53.842128 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.842119 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 20:06:53.842420 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.842402 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-st85q"] Nov 01 20:06:53.921751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.921726 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck5bx\" (UniqueName: \"kubernetes.io/projected/bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96-kube-api-access-ck5bx\") pod \"jobset-operator-674cd59dc5-st85q\" (UID: \"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:53.921876 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.921762 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96-tmp\") pod \"jobset-operator-674cd59dc5-st85q\" (UID: \"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:53.922147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.922127 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96-tmp\") pod \"jobset-operator-674cd59dc5-st85q\" (UID: \"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:53.989038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:53.988982 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck5bx\" (UniqueName: \"kubernetes.io/projected/bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96-kube-api-access-ck5bx\") pod \"jobset-operator-674cd59dc5-st85q\" (UID: \"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:54.055928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:54.055908 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" Nov 01 20:06:54.425698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:54.425668 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-st85q"] Nov 01 20:06:54.429835 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:06:54.429812 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf4a47cf_a8f4_43d9_a8af_fdda5bc06b96.slice/crio-1a45bc1df56473ccf994ae9a2e9f53928d90c72447b4c3b9551af6efe2ebf7f2 WatchSource:0}: Error finding container 1a45bc1df56473ccf994ae9a2e9f53928d90c72447b4c3b9551af6efe2ebf7f2: Status 404 returned error can't find the container with id 1a45bc1df56473ccf994ae9a2e9f53928d90c72447b4c3b9551af6efe2ebf7f2 Nov 01 20:06:54.906451 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:54.906423 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" event={"ID":"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96","Type":"ContainerStarted","Data":"1a45bc1df56473ccf994ae9a2e9f53928d90c72447b4c3b9551af6efe2ebf7f2"} Nov 01 20:06:55.878811 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:06:55.878783 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d77b8c7cf-mshvq" Nov 01 20:07:00.032692 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:07:00.032660 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:07:01.930864 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:01.930783 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" event={"ID":"bf4a47cf-a8f4-43d9-a8af-fdda5bc06b96","Type":"ContainerStarted","Data":"a067ef71b69c48d8a8f145d102aa4c00162971e316b8e7108f001d4c6f007d31"} Nov 01 20:07:02.045548 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:02.045491 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-st85q" podStartSLOduration=1.9045090999999998 podStartE2EDuration="9.045478096s" totalImagesPullingTime="7.140968996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:02.045070207 +0000 UTC m=+222.989908075" watchObservedRunningTime="2026-11-01 20:07:02.045478096 +0000 UTC m=+222.990315963" Nov 01 20:07:03.519334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.519304 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68"] Nov 01 20:07:03.522407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.522392 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.592466 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.592434 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c283462-a39d-406e-be48-ad3c33a27857-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.592466 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.592465 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4nq\" (UniqueName: \"kubernetes.io/projected/1c283462-a39d-406e-be48-ad3c33a27857-kube-api-access-tr4nq\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.592649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.592495 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1c283462-a39d-406e-be48-ad3c33a27857-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.624175 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.624147 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:07:03.624308 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.624238 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 20:07:03.624308 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.624248 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z5b7z\"" Nov 01 20:07:03.624398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.624314 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:07:03.626741 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.626724 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 20:07:03.627302 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.627285 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68"] Nov 01 20:07:03.693127 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.693105 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1c283462-a39d-406e-be48-ad3c33a27857-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.693243 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.693159 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c283462-a39d-406e-be48-ad3c33a27857-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.693308 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.693277 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4nq\" (UniqueName: \"kubernetes.io/projected/1c283462-a39d-406e-be48-ad3c33a27857-kube-api-access-tr4nq\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.693822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.693800 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1c283462-a39d-406e-be48-ad3c33a27857-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.695523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.695505 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c283462-a39d-406e-be48-ad3c33a27857-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.749702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.749680 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4nq\" (UniqueName: \"kubernetes.io/projected/1c283462-a39d-406e-be48-ad3c33a27857-kube-api-access-tr4nq\") pod \"kuadrant-console-plugin-66c4fb575d-vnw68\" (UID: \"1c283462-a39d-406e-be48-ad3c33a27857\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:03.832613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:03.832560 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" Nov 01 20:07:04.202702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:04.202619 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68"] Nov 01 20:07:04.206034 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:07:04.206008 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c283462_a39d_406e_be48_ad3c33a27857.slice/crio-5baa348d7906d4f2b716266564deffad454f724634d9214a03d61651f959e264 WatchSource:0}: Error finding container 5baa348d7906d4f2b716266564deffad454f724634d9214a03d61651f959e264: Status 404 returned error can't find the container with id 5baa348d7906d4f2b716266564deffad454f724634d9214a03d61651f959e264 Nov 01 20:07:04.941344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:04.941306 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" event={"ID":"1c283462-a39d-406e-be48-ad3c33a27857","Type":"ContainerStarted","Data":"5baa348d7906d4f2b716266564deffad454f724634d9214a03d61651f959e264"} Nov 01 20:07:08.688512 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.687416 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn"] Nov 01 20:07:08.690606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.690583 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.744173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.744145 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 20:07:08.744414 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.744214 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-db7jz\"" Nov 01 20:07:08.791561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.791537 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 20:07:08.792436 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.792416 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn"] Nov 01 20:07:08.832355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.832329 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-tls-certs\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.832481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.832365 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-tmp\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.832481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.832387 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpmm\" (UniqueName: \"kubernetes.io/projected/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-kube-api-access-knpmm\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.933381 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.933341 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-tls-certs\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.933588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.933406 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-tmp\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.933588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.933445 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpmm\" (UniqueName: \"kubernetes.io/projected/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-kube-api-access-knpmm\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.936264 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.936234 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-tmp\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.939023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.938983 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-tls-certs\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:08.989572 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:08.989547 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpmm\" (UniqueName: \"kubernetes.io/projected/081918c5-3f3d-4d01-aafe-b8f8ce741bfd-kube-api-access-knpmm\") pod \"kube-auth-proxy-5c84dcdb75-q8bgn\" (UID: \"081918c5-3f3d-4d01-aafe-b8f8ce741bfd\") " pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:09.002358 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:09.002335 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" Nov 01 20:07:09.370616 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:09.370581 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn"] Nov 01 20:07:09.375005 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:07:09.374978 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081918c5_3f3d_4d01_aafe_b8f8ce741bfd.slice/crio-668fd013ba12d45e4a5aa1fb3e4199d62715fb69d4cd21302feaee47f35c092e WatchSource:0}: Error finding container 668fd013ba12d45e4a5aa1fb3e4199d62715fb69d4cd21302feaee47f35c092e: Status 404 returned error can't find the container with id 668fd013ba12d45e4a5aa1fb3e4199d62715fb69d4cd21302feaee47f35c092e Nov 01 20:07:09.959524 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:09.959492 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" event={"ID":"081918c5-3f3d-4d01-aafe-b8f8ce741bfd","Type":"ContainerStarted","Data":"668fd013ba12d45e4a5aa1fb3e4199d62715fb69d4cd21302feaee47f35c092e"} Nov 01 20:07:13.976485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:13.976441 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" event={"ID":"1c283462-a39d-406e-be48-ad3c33a27857","Type":"ContainerStarted","Data":"a32c15806d9144dfa6cfbaa42165de67e78341a1dab8284f5937625cd683195c"} Nov 01 20:07:13.977862 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:13.977842 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" event={"ID":"081918c5-3f3d-4d01-aafe-b8f8ce741bfd","Type":"ContainerStarted","Data":"2db240c3c932985b89d09fdf7148d27cc8f0d0036dde39ba4db6f605e071e555"} Nov 01 20:07:14.089535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:14.089485 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vnw68" podStartSLOduration=2.119033816 podStartE2EDuration="11.089472323s" totalImagesPullingTime="8.970438507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:14.089054338 +0000 UTC m=+235.033892206" watchObservedRunningTime="2026-11-01 20:07:14.089472323 +0000 UTC m=+235.034310189" Nov 01 20:07:14.242795 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:14.242621 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c84dcdb75-q8bgn" podStartSLOduration=2.654885395 podStartE2EDuration="6.242600791s" totalImagesPullingTime="3.587715396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:14.241828933 +0000 UTC m=+235.186666798" watchObservedRunningTime="2026-11-01 20:07:14.242600791 +0000 UTC m=+235.187438661" Nov 01 20:07:15.666870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.666828 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zjxvf"] Nov 01 20:07:15.672101 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.672080 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:07:15.726823 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.726788 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zjxvf"] Nov 01 20:07:15.775343 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.775324 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dt6v\"" Nov 01 20:07:15.788129 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.788102 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2x5m\" (UniqueName: \"kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m\") pod \"authorino-56c6c9ddd5-zjxvf\" (UID: \"76e42592-0dd4-4213-9c3d-d8acaaa642cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:07:15.888905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.888873 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2x5m\" (UniqueName: \"kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m\") pod \"authorino-56c6c9ddd5-zjxvf\" (UID: \"76e42592-0dd4-4213-9c3d-d8acaaa642cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:07:15.945721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.945620 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2x5m\" (UniqueName: \"kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m\") pod \"authorino-56c6c9ddd5-zjxvf\" (UID: \"76e42592-0dd4-4213-9c3d-d8acaaa642cc\") " pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:07:15.981280 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:15.981249 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:07:16.344267 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:16.344232 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zjxvf"] Nov 01 20:07:16.347785 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:07:16.347760 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e42592_0dd4_4213_9c3d_d8acaaa642cc.slice/crio-401d6d4947ffbbe85ce92e8a225ab183662e8380733fbaf3086d5487add141d6 WatchSource:0}: Error finding container 401d6d4947ffbbe85ce92e8a225ab183662e8380733fbaf3086d5487add141d6: Status 404 returned error can't find the container with id 401d6d4947ffbbe85ce92e8a225ab183662e8380733fbaf3086d5487add141d6 Nov 01 20:07:16.988434 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:16.988405 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" event={"ID":"76e42592-0dd4-4213-9c3d-d8acaaa642cc","Type":"ContainerStarted","Data":"401d6d4947ffbbe85ce92e8a225ab183662e8380733fbaf3086d5487add141d6"} Nov 01 20:07:20.001698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:20.001664 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" event={"ID":"76e42592-0dd4-4213-9c3d-d8acaaa642cc","Type":"ContainerStarted","Data":"9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc"} Nov 01 20:07:20.116753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:20.116660 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" podStartSLOduration=1.689126862 podStartE2EDuration="5.116629368s" totalImagesPullingTime="3.427502506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:20.115735969 +0000 UTC m=+241.060573834" watchObservedRunningTime="2026-11-01 20:07:20.116629368 +0000 UTC m=+241.061467238" Nov 01 20:07:30.035607 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:07:30.035566 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:07:39.558056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.558016 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg"] Nov 01 20:07:39.566145 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.566122 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.618591 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.618561 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg"] Nov 01 20:07:39.626609 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.626590 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:07:39.658028 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.658001 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thnsj\" (UniqueName: \"kubernetes.io/projected/2f54063d-c15d-4866-988b-a38a0a839dad-kube-api-access-thnsj\") pod \"postgres-7cdbf6dc44-5nrpg\" (UID: \"2f54063d-c15d-4866-988b-a38a0a839dad\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.658136 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.658050 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2f54063d-c15d-4866-988b-a38a0a839dad-data\") pod \"postgres-7cdbf6dc44-5nrpg\" (UID: \"2f54063d-c15d-4866-988b-a38a0a839dad\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.667340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.667324 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:07:39.698845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.698823 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 20:07:39.699032 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.698969 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fbmgz\"" Nov 01 20:07:39.758930 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.758907 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thnsj\" (UniqueName: \"kubernetes.io/projected/2f54063d-c15d-4866-988b-a38a0a839dad-kube-api-access-thnsj\") pod \"postgres-7cdbf6dc44-5nrpg\" (UID: \"2f54063d-c15d-4866-988b-a38a0a839dad\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.759022 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.758956 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2f54063d-c15d-4866-988b-a38a0a839dad-data\") pod \"postgres-7cdbf6dc44-5nrpg\" (UID: \"2f54063d-c15d-4866-988b-a38a0a839dad\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.759258 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.759238 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2f54063d-c15d-4866-988b-a38a0a839dad-data\") pod \"postgres-7cdbf6dc44-5nrpg\" (UID: \"2f54063d-c15d-4866-988b-a38a0a839dad\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.828860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.828816 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thnsj\" (UniqueName: \"kubernetes.io/projected/2f54063d-c15d-4866-988b-a38a0a839dad-kube-api-access-thnsj\") pod \"postgres-7cdbf6dc44-5nrpg\" (UID: \"2f54063d-c15d-4866-988b-a38a0a839dad\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:39.879383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:39.879347 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:40.244468 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:40.244401 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg"] Nov 01 20:07:40.248498 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:07:40.248463 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f54063d_c15d_4866_988b_a38a0a839dad.slice/crio-a295e9cc7914a03c2d73fcd741961690c3a3c3fbb97e29f0d73086a3800e375f WatchSource:0}: Error finding container a295e9cc7914a03c2d73fcd741961690c3a3c3fbb97e29f0d73086a3800e375f: Status 404 returned error can't find the container with id a295e9cc7914a03c2d73fcd741961690c3a3c3fbb97e29f0d73086a3800e375f Nov 01 20:07:41.076348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:41.076312 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" event={"ID":"2f54063d-c15d-4866-988b-a38a0a839dad","Type":"ContainerStarted","Data":"a295e9cc7914a03c2d73fcd741961690c3a3c3fbb97e29f0d73086a3800e375f"} Nov 01 20:07:47.102097 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:47.102020 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" event={"ID":"2f54063d-c15d-4866-988b-a38a0a839dad","Type":"ContainerStarted","Data":"7d90be4876c1e69ffe92f77c89fad57ec7e28005f02b2a714e8b64ae3269002b"} Nov 01 20:07:47.102478 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:47.102157 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:07:47.215322 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:47.215257 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" podStartSLOduration=1.662272689 podStartE2EDuration="8.215238603s" totalImagesPullingTime="6.552965914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:47.214606734 +0000 UTC m=+268.159444603" watchObservedRunningTime="2026-11-01 20:07:47.215238603 +0000 UTC m=+268.160076472" Nov 01 20:07:53.134168 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:07:53.134083 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nrpg" Nov 01 20:08:00.033984 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:08:00.033947 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:08:21.286112 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.286077 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm"] Nov 01 20:08:21.287986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.287970 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.329827 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.329793 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.330003 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.329840 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.330003 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.329939 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.330003 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.329971 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4x52\" (UniqueName: \"kubernetes.io/projected/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-kube-api-access-t4x52\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.330139 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.330006 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.330139 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.330125 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.330206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.330155 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.394580 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.394559 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm"] Nov 01 20:08:21.431402 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431378 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.431496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431410 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.431496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431450 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.431496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431482 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4x52\" (UniqueName: \"kubernetes.io/projected/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-kube-api-access-t4x52\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.431632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431517 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.431632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431545 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.431632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.431571 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.432286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.432264 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.433856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.433837 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.433856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.433846 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.434145 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.434128 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.434229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.434210 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.492062 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.492037 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.493110 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.493085 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4x52\" (UniqueName: \"kubernetes.io/projected/6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31-kube-api-access-t4x52\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlppm\" (UID: \"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:21.597564 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:21.597492 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:22.017078 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.017046 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm"] Nov 01 20:08:22.020907 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:08:22.020875 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9f6b66_47d6_45b9_bd5d_4d36a72a6c31.slice/crio-5a86aef7dcede7f1bed830e8dfe2fcd3a94ec6b30f6ca90c51f270eb9aaeab99 WatchSource:0}: Error finding container 5a86aef7dcede7f1bed830e8dfe2fcd3a94ec6b30f6ca90c51f270eb9aaeab99: Status 404 returned error can't find the container with id 5a86aef7dcede7f1bed830e8dfe2fcd3a94ec6b30f6ca90c51f270eb9aaeab99 Nov 01 20:08:22.022886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.022857 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:22.022944 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.022921 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:22.220575 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.220539 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" event={"ID":"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31","Type":"ContainerStarted","Data":"f4a0ef02d8493dac5bd005de6461f87423092c3608364fa79baa9557193120cd"} Nov 01 20:08:22.220766 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.220592 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" event={"ID":"6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31","Type":"ContainerStarted","Data":"5a86aef7dcede7f1bed830e8dfe2fcd3a94ec6b30f6ca90c51f270eb9aaeab99"} Nov 01 20:08:22.220766 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.220662 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:22.338489 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:22.338380 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" podStartSLOduration=1.338360772 podStartE2EDuration="1.338360772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:22.336760561 +0000 UTC m=+303.281598442" watchObservedRunningTime="2026-11-01 20:08:22.338360772 +0000 UTC m=+303.283198644" Nov 01 20:08:23.225972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:23.225935 5444 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hlppm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:08:23.226141 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:23.226018 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" podUID="6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:08:26.226156 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.226121 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlppm" Nov 01 20:08:26.442306 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.442270 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw"] Nov 01 20:08:26.442497 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.442453 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" podUID="e6c2061f-2923-4a25-81ff-761a97ed57f4" containerName="discovery" containerID="cri-o://c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372" gracePeriod=30 Nov 01 20:08:26.690293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.690270 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:08:26.773749 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773705 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.773749 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773746 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.773986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773776 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.773986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773803 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.773986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773865 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.773986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773894 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.773986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.773925 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token\") pod \"e6c2061f-2923-4a25-81ff-761a97ed57f4\" (UID: \"e6c2061f-2923-4a25-81ff-761a97ed57f4\") " Nov 01 20:08:26.774237 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.774192 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:08:26.776768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.776716 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:08:26.776903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.776776 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:08:26.776903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.776792 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:08:26.776903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.776889 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:08:26.777019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.776907 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "kube-api-access-qvs79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:08:26.777019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.776964 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token" pod "e6c2061f-2923-4a25-81ff-761a97ed57f4" (UID: "e6c2061f-2923-4a25-81ff-761a97ed57f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874486 5444 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-dns-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874514 5444 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-csr-ca-configmap\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874525 5444 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-kubeconfig\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874535 5444 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c2061f-2923-4a25-81ff-761a97ed57f4-local-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874545 5444 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6c2061f-2923-4a25-81ff-761a97ed57f4-cacerts\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874553 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvs79\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-kube-api-access-qvs79\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:26.874561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:26.874562 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c2061f-2923-4a25-81ff-761a97ed57f4-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:08:27.240980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.240942 5444 generic.go:356] "Generic (PLEG): container finished" podID="e6c2061f-2923-4a25-81ff-761a97ed57f4" containerID="c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372" exitCode=0 Nov 01 20:08:27.241410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.240986 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" Nov 01 20:08:27.241410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.240987 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" event={"ID":"e6c2061f-2923-4a25-81ff-761a97ed57f4","Type":"ContainerDied","Data":"c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372"} Nov 01 20:08:27.241410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.241114 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw" event={"ID":"e6c2061f-2923-4a25-81ff-761a97ed57f4","Type":"ContainerDied","Data":"bd7fe8e2fc8526e43bebea5031be328573c6811b954b480a6f2e64655742d9cf"} Nov 01 20:08:27.241410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.241138 5444 scope.go:122] "RemoveContainer" containerID="c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372" Nov 01 20:08:27.249305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.249284 5444 scope.go:122] "RemoveContainer" containerID="c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372" Nov 01 20:08:27.249589 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:08:27.249571 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372\": container with ID starting with c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372 not found: ID does not exist" containerID="c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372" Nov 01 20:08:27.249649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.249599 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372"} err="failed to get container status \"c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372\": rpc error: code = NotFound desc = could not find container \"c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372\": container with ID starting with c36cf653d3ceede40f0fc8a4240517f4a0d72074b4a38f41383878fa5e0a3372 not found: ID does not exist" Nov 01 20:08:27.410703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.410669 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw"] Nov 01 20:08:27.414880 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.414791 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8wtjw"] Nov 01 20:08:27.561532 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.561498 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56"] Nov 01 20:08:27.561955 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.561937 5444 cpu_manager.go:414] "containerMap: removing container" podUID="e6c2061f-2923-4a25-81ff-761a97ed57f4" containerName="discovery" Nov 01 20:08:27.562024 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.561960 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c2061f-2923-4a25-81ff-761a97ed57f4" containerName="discovery" Nov 01 20:08:27.562087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.562075 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c2061f-2923-4a25-81ff-761a97ed57f4" containerName="discovery" Nov 01 20:08:27.566192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.566170 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.618265 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.618233 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gvxsl\"" Nov 01 20:08:27.618456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.618302 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 20:08:27.618703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.618687 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 20:08:27.618703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.618698 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 20:08:27.622664 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.622613 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56"] Nov 01 20:08:27.682435 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.682338 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj5h\" (UniqueName: \"kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h\") pod \"llmisvc-controller-manager-869d5dcc5d-6tj56\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.682672 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.682508 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6tj56\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.783976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.783933 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6tj56\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.784166 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.783985 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj5h\" (UniqueName: \"kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h\") pod \"llmisvc-controller-manager-869d5dcc5d-6tj56\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.786460 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.786437 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-6tj56\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.840547 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.840506 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj5h\" (UniqueName: \"kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h\") pod \"llmisvc-controller-manager-869d5dcc5d-6tj56\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:27.879437 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:27.879398 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:28.251470 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:28.251440 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56"] Nov 01 20:08:28.255052 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:08:28.255024 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod63f01f1f_5bda_4f30_907d_55afd9c9b947.slice/crio-a4bd8f8dce62f93c013b42c41d564e2b45f1ad7d3e2154b46b5aef926e24e696 WatchSource:0}: Error finding container a4bd8f8dce62f93c013b42c41d564e2b45f1ad7d3e2154b46b5aef926e24e696: Status 404 returned error can't find the container with id a4bd8f8dce62f93c013b42c41d564e2b45f1ad7d3e2154b46b5aef926e24e696 Nov 01 20:08:29.157069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:29.157027 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c2061f-2923-4a25-81ff-761a97ed57f4" path="/var/lib/kubelet/pods/e6c2061f-2923-4a25-81ff-761a97ed57f4/volumes" Nov 01 20:08:29.251227 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:29.251187 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" event={"ID":"63f01f1f-5bda-4f30-907d-55afd9c9b947","Type":"ContainerStarted","Data":"a4bd8f8dce62f93c013b42c41d564e2b45f1ad7d3e2154b46b5aef926e24e696"} Nov 01 20:08:30.037395 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:08:30.037352 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:08:31.262627 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:31.262583 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" event={"ID":"63f01f1f-5bda-4f30-907d-55afd9c9b947","Type":"ContainerStarted","Data":"864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e"} Nov 01 20:08:31.263122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:31.262767 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:08:31.378400 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:31.378279 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" podStartSLOduration=1.9013831730000001 podStartE2EDuration="4.378262042s" totalImagesPullingTime="2.476878869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:31.377771218 +0000 UTC m=+312.322609085" watchObservedRunningTime="2026-11-01 20:08:31.378262042 +0000 UTC m=+312.323099908" Nov 01 20:08:43.934316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:43.934283 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8"] Nov 01 20:08:43.937870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:43.937855 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.011294 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.011257 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.011294 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.011298 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.011539 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.011320 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.011608 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.011534 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.039483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.039452 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s5dbl\"" Nov 01 20:08:44.039483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.039452 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 20:08:44.039678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.039541 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 20:08:44.040289 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.040269 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8"] Nov 01 20:08:44.112507 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.112476 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.112728 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.112511 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.112728 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.112535 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.112728 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.112585 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.113197 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.113173 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.115118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.115086 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.115203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.115086 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.177302 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.177275 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg\") pod \"maas-api-848c4dd8d8-8kwr8\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.249665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.249571 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:44.618245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:44.618207 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8"] Nov 01 20:08:44.621932 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:08:44.621903 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdd09e9_6db5_4a25_aac1_41c4ee570ec5.slice/crio-a68864eb653300822467a8f8c7b8d4dfa136211b2d656e0ad7102aaf9fef7a30 WatchSource:0}: Error finding container a68864eb653300822467a8f8c7b8d4dfa136211b2d656e0ad7102aaf9fef7a30: Status 404 returned error can't find the container with id a68864eb653300822467a8f8c7b8d4dfa136211b2d656e0ad7102aaf9fef7a30 Nov 01 20:08:45.317860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:45.317821 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" event={"ID":"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5","Type":"ContainerStarted","Data":"a68864eb653300822467a8f8c7b8d4dfa136211b2d656e0ad7102aaf9fef7a30"} Nov 01 20:08:52.347878 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:52.347838 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" event={"ID":"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5","Type":"ContainerStarted","Data":"ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da"} Nov 01 20:08:52.348341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:52.347939 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:52.461648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:52.461588 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" podStartSLOduration=2.76717859 podStartE2EDuration="9.461575691s" totalImagesPullingTime="6.694397101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:52.460770151 +0000 UTC m=+333.405608018" watchObservedRunningTime="2026-11-01 20:08:52.461575691 +0000 UTC m=+333.406413560" Nov 01 20:08:58.357750 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:58.357719 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:08:59.310413 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.310376 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7"] Nov 01 20:08:59.313916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.313895 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.416312 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.416278 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7"] Nov 01 20:08:59.439122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.439092 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdghj\" (UniqueName: \"kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.439271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.439128 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.439271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.439156 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.439271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.439220 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.540460 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.540425 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.540665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.540490 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdghj\" (UniqueName: \"kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.540665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.540510 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.540665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.540540 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.541202 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.541179 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.543033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.543010 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.543150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.543050 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.614614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.614539 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdghj\" (UniqueName: \"kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj\") pod \"maas-api-845db7b4d-d4dc7\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.625348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.625320 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:08:59.994950 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:08:59.994868 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7"] Nov 01 20:09:00.036143 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:09:00.036108 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:09:00.378344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:00.378308 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" event={"ID":"c9b1cc77-92f0-4160-8f97-27d1e736ef46","Type":"ContainerStarted","Data":"ac9fe528c45694d8f51183be173e0cc7c28bd41fd806bf7c940c8fc02ef133bb"} Nov 01 20:09:01.383596 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:01.383562 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" event={"ID":"c9b1cc77-92f0-4160-8f97-27d1e736ef46","Type":"ContainerStarted","Data":"ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92"} Nov 01 20:09:01.384000 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:01.383702 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:09:01.497812 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:01.497760 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" podStartSLOduration=1.681914499 podStartE2EDuration="2.497745022s" totalImagesPullingTime="815.830523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:01.496628778 +0000 UTC m=+342.441466640" watchObservedRunningTime="2026-11-01 20:09:01.497745022 +0000 UTC m=+342.442582890" Nov 01 20:09:02.268770 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:02.268737 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:09:03.385460 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:03.385426 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zjxvf"] Nov 01 20:09:03.386203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:03.386166 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" podUID="76e42592-0dd4-4213-9c3d-d8acaaa642cc" containerName="authorino" containerID="cri-o://9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc" gracePeriod=30 Nov 01 20:09:03.631119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:03.631092 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:09:03.778866 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:03.778833 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m\" (UniqueName: \"kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m\") pod \"76e42592-0dd4-4213-9c3d-d8acaaa642cc\" (UID: \"76e42592-0dd4-4213-9c3d-d8acaaa642cc\") " Nov 01 20:09:03.780979 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:03.780950 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m" pod "76e42592-0dd4-4213-9c3d-d8acaaa642cc" (UID: "76e42592-0dd4-4213-9c3d-d8acaaa642cc"). InnerVolumeSpecName "kube-api-access-c2x5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:09:03.880257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:03.880218 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2x5m\" (UniqueName: \"kubernetes.io/projected/76e42592-0dd4-4213-9c3d-d8acaaa642cc-kube-api-access-c2x5m\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:04.396398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.396359 5444 generic.go:356] "Generic (PLEG): container finished" podID="76e42592-0dd4-4213-9c3d-d8acaaa642cc" containerID="9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc" exitCode=0 Nov 01 20:09:04.396835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.396396 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" event={"ID":"76e42592-0dd4-4213-9c3d-d8acaaa642cc","Type":"ContainerDied","Data":"9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc"} Nov 01 20:09:04.396835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.396442 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" event={"ID":"76e42592-0dd4-4213-9c3d-d8acaaa642cc","Type":"ContainerDied","Data":"401d6d4947ffbbe85ce92e8a225ab183662e8380733fbaf3086d5487add141d6"} Nov 01 20:09:04.396835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.396463 5444 scope.go:122] "RemoveContainer" containerID="9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc" Nov 01 20:09:04.396835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.396405 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zjxvf" Nov 01 20:09:04.404606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.404586 5444 scope.go:122] "RemoveContainer" containerID="9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc" Nov 01 20:09:04.404882 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:09:04.404863 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc\": container with ID starting with 9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc not found: ID does not exist" containerID="9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc" Nov 01 20:09:04.404931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.404895 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc"} err="failed to get container status \"9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc\": rpc error: code = NotFound desc = could not find container \"9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc\": container with ID starting with 9fdecb88fb5d3aca2d1b69331d1ebe2f0f645befda00ae4573afb5f6c05ea1bc not found: ID does not exist" Nov 01 20:09:04.556699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.556662 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zjxvf"] Nov 01 20:09:04.558568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:04.558540 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zjxvf"] Nov 01 20:09:05.156771 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:05.156737 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e42592-0dd4-4213-9c3d-d8acaaa642cc" path="/var/lib/kubelet/pods/76e42592-0dd4-4213-9c3d-d8acaaa642cc/volumes" Nov 01 20:09:07.392691 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.392662 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 20:09:07.556379 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.556347 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8"] Nov 01 20:09:07.556622 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.556598 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" podUID="3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" containerName="maas-api" containerID="cri-o://ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da" gracePeriod=30 Nov 01 20:09:07.800838 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.800808 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:09:07.809351 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.809328 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs\") pod \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " Nov 01 20:09:07.809472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.809361 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca\") pod \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " Nov 01 20:09:07.809472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.809415 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls\") pod \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " Nov 01 20:09:07.809472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.809436 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg\") pod \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\" (UID: \"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5\") " Nov 01 20:09:07.809774 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.809746 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca" pod "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" (UID: "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:09:07.811631 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.811601 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs" pod "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" (UID: "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:09:07.811735 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.811663 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls" pod "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" (UID: "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:09:07.811735 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.811661 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg" pod "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" (UID: "3fdd09e9-6db5-4a25-aac1-41c4ee570ec5"). InnerVolumeSpecName "kube-api-access-t8scg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:09:07.910742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.910709 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:07.910742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.910740 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8scg\" (UniqueName: \"kubernetes.io/projected/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-kube-api-access-t8scg\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:07.910742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.910751 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:07.910998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:07.910760 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:09:08.414527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.414493 5444 generic.go:356] "Generic (PLEG): container finished" podID="3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" containerID="ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da" exitCode=0 Nov 01 20:09:08.415023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.414542 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" Nov 01 20:09:08.415023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.414573 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" event={"ID":"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5","Type":"ContainerDied","Data":"ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da"} Nov 01 20:09:08.415023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.414624 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8" event={"ID":"3fdd09e9-6db5-4a25-aac1-41c4ee570ec5","Type":"ContainerDied","Data":"a68864eb653300822467a8f8c7b8d4dfa136211b2d656e0ad7102aaf9fef7a30"} Nov 01 20:09:08.415023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.414675 5444 scope.go:122] "RemoveContainer" containerID="ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da" Nov 01 20:09:08.423439 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.423425 5444 scope.go:122] "RemoveContainer" containerID="ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da" Nov 01 20:09:08.423716 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:09:08.423695 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da\": container with ID starting with ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da not found: ID does not exist" containerID="ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da" Nov 01 20:09:08.423772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.423726 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da"} err="failed to get container status \"ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da\": rpc error: code = NotFound desc = could not find container \"ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da\": container with ID starting with ddca29772c49c7852afb8d81c2bf4d9031c36cc4feaada59e9fa9790628c61da not found: ID does not exist" Nov 01 20:09:08.575773 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.575734 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8"] Nov 01 20:09:08.625340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:08.625308 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-848c4dd8d8-8kwr8"] Nov 01 20:09:09.157806 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:09.157778 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" path="/var/lib/kubelet/pods/3fdd09e9-6db5-4a25-aac1-41c4ee570ec5/volumes" Nov 01 20:09:30.033937 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:09:30.033891 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:09:44.590981 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.590930 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qh45n"] Nov 01 20:09:44.591407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.591360 5444 cpu_manager.go:414] "containerMap: removing container" podUID="76e42592-0dd4-4213-9c3d-d8acaaa642cc" containerName="authorino" Nov 01 20:09:44.591407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.591376 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e42592-0dd4-4213-9c3d-d8acaaa642cc" containerName="authorino" Nov 01 20:09:44.591407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.591390 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" containerName="maas-api" Nov 01 20:09:44.591407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.591396 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" containerName="maas-api" Nov 01 20:09:44.591540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.591451 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e42592-0dd4-4213-9c3d-d8acaaa642cc" containerName="authorino" Nov 01 20:09:44.591540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.591460 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdd09e9-6db5-4a25-aac1-41c4ee570ec5" containerName="maas-api" Nov 01 20:09:44.595475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.595460 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.652429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.652398 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qh45n"] Nov 01 20:09:44.692928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.692895 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88630e47-4588-44a1-94ac-f39973ad1406-cert\") pod \"llmisvc-controller-manager-6466c554d8-qh45n\" (UID: \"88630e47-4588-44a1-94ac-f39973ad1406\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.693089 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.692946 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7vf\" (UniqueName: \"kubernetes.io/projected/88630e47-4588-44a1-94ac-f39973ad1406-kube-api-access-pz7vf\") pod \"llmisvc-controller-manager-6466c554d8-qh45n\" (UID: \"88630e47-4588-44a1-94ac-f39973ad1406\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.793900 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.793862 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88630e47-4588-44a1-94ac-f39973ad1406-cert\") pod \"llmisvc-controller-manager-6466c554d8-qh45n\" (UID: \"88630e47-4588-44a1-94ac-f39973ad1406\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.794075 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.793912 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7vf\" (UniqueName: \"kubernetes.io/projected/88630e47-4588-44a1-94ac-f39973ad1406-kube-api-access-pz7vf\") pod \"llmisvc-controller-manager-6466c554d8-qh45n\" (UID: \"88630e47-4588-44a1-94ac-f39973ad1406\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.796251 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.796223 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88630e47-4588-44a1-94ac-f39973ad1406-cert\") pod \"llmisvc-controller-manager-6466c554d8-qh45n\" (UID: \"88630e47-4588-44a1-94ac-f39973ad1406\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.850674 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.850586 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7vf\" (UniqueName: \"kubernetes.io/projected/88630e47-4588-44a1-94ac-f39973ad1406-kube-api-access-pz7vf\") pod \"llmisvc-controller-manager-6466c554d8-qh45n\" (UID: \"88630e47-4588-44a1-94ac-f39973ad1406\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:44.908098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:44.908060 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:45.277826 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:45.277800 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qh45n"] Nov 01 20:09:45.281222 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:09:45.281198 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88630e47_4588_44a1_94ac_f39973ad1406.slice/crio-4867935e1f6cceeda412aeccfc19949e1c307a0d12dbd52b63d10c2daddce34a WatchSource:0}: Error finding container 4867935e1f6cceeda412aeccfc19949e1c307a0d12dbd52b63d10c2daddce34a: Status 404 returned error can't find the container with id 4867935e1f6cceeda412aeccfc19949e1c307a0d12dbd52b63d10c2daddce34a Nov 01 20:09:45.560649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:45.560548 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" event={"ID":"88630e47-4588-44a1-94ac-f39973ad1406","Type":"ContainerStarted","Data":"4867935e1f6cceeda412aeccfc19949e1c307a0d12dbd52b63d10c2daddce34a"} Nov 01 20:09:46.566001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:46.565964 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" event={"ID":"88630e47-4588-44a1-94ac-f39973ad1406","Type":"ContainerStarted","Data":"3b076d837a7857cc2f2ce7989ab1c55042b1bcaca51bc8cf8e88f836d08e2e2a"} Nov 01 20:09:46.566425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:46.566090 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:09:46.680273 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:09:46.680218 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" podStartSLOduration=1.8722003900000002 podStartE2EDuration="2.680191781s" totalImagesPullingTime="807.991391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:46.679234154 +0000 UTC m=+387.624072022" watchObservedRunningTime="2026-11-01 20:09:46.680191781 +0000 UTC m=+387.625029650" Nov 01 20:10:00.033850 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:10:00.033814 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:10:17.571408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:17.571376 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qh45n" Nov 01 20:10:17.743571 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:17.743536 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56"] Nov 01 20:10:17.743881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:17.743828 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" podUID="63f01f1f-5bda-4f30-907d-55afd9c9b947" containerName="manager" containerID="cri-o://864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e" gracePeriod=30 Nov 01 20:10:17.985215 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:17.985193 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:10:18.056596 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.056560 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h\" (UniqueName: \"kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h\") pod \"63f01f1f-5bda-4f30-907d-55afd9c9b947\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " Nov 01 20:10:18.056814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.056613 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert\" (UniqueName: \"kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert\") pod \"63f01f1f-5bda-4f30-907d-55afd9c9b947\" (UID: \"63f01f1f-5bda-4f30-907d-55afd9c9b947\") " Nov 01 20:10:18.058820 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.058789 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert" pod "63f01f1f-5bda-4f30-907d-55afd9c9b947" (UID: "63f01f1f-5bda-4f30-907d-55afd9c9b947"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:10:18.058953 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.058844 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h" pod "63f01f1f-5bda-4f30-907d-55afd9c9b947" (UID: "63f01f1f-5bda-4f30-907d-55afd9c9b947"). InnerVolumeSpecName "kube-api-access-nkj5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:10:18.157680 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.157573 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkj5h\" (UniqueName: \"kubernetes.io/projected/63f01f1f-5bda-4f30-907d-55afd9c9b947-kube-api-access-nkj5h\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:10:18.157680 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.157605 5444 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63f01f1f-5bda-4f30-907d-55afd9c9b947-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:10:18.686279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.686249 5444 generic.go:356] "Generic (PLEG): container finished" podID="63f01f1f-5bda-4f30-907d-55afd9c9b947" containerID="864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e" exitCode=0 Nov 01 20:10:18.686785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.686295 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" Nov 01 20:10:18.686785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.686304 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" event={"ID":"63f01f1f-5bda-4f30-907d-55afd9c9b947","Type":"ContainerDied","Data":"864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e"} Nov 01 20:10:18.686785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.686344 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56" event={"ID":"63f01f1f-5bda-4f30-907d-55afd9c9b947","Type":"ContainerDied","Data":"a4bd8f8dce62f93c013b42c41d564e2b45f1ad7d3e2154b46b5aef926e24e696"} Nov 01 20:10:18.686785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.686364 5444 scope.go:122] "RemoveContainer" containerID="864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e" Nov 01 20:10:18.696192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.696175 5444 scope.go:122] "RemoveContainer" containerID="864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e" Nov 01 20:10:18.696506 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:10:18.696487 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e\": container with ID starting with 864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e not found: ID does not exist" containerID="864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e" Nov 01 20:10:18.696572 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.696514 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e"} err="failed to get container status \"864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e\": rpc error: code = NotFound desc = could not find container \"864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e\": container with ID starting with 864f621392a06779dbd13c04b120f5761a634467831f31f7a2995b1afa96338e not found: ID does not exist" Nov 01 20:10:18.848085 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.848040 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56"] Nov 01 20:10:18.897465 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:18.897424 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-6tj56"] Nov 01 20:10:19.157483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:10:19.157446 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f01f1f-5bda-4f30-907d-55afd9c9b947" path="/var/lib/kubelet/pods/63f01f1f-5bda-4f30-907d-55afd9c9b947/volumes" Nov 01 20:10:30.034692 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:10:30.034653 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:11:00.036029 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:11:00.035954 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:11:30.034541 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:11:30.034502 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:11:39.161234 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.161205 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx"] Nov 01 20:11:39.161686 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.161480 5444 cpu_manager.go:414] "containerMap: removing container" podUID="63f01f1f-5bda-4f30-907d-55afd9c9b947" containerName="manager" Nov 01 20:11:39.161686 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.161491 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f01f1f-5bda-4f30-907d-55afd9c9b947" containerName="manager" Nov 01 20:11:39.161686 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.161561 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f01f1f-5bda-4f30-907d-55afd9c9b947" containerName="manager" Nov 01 20:11:39.164237 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.164220 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.216287 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.216261 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bm6m7\"" Nov 01 20:11:39.285421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285382 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44ht\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285425 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285656 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285442 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285656 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285499 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285656 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285588 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285656 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285609 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285656 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285627 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285683 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.285843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.285732 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.373337 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.373280 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx"] Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388246 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44ht\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388308 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388342 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388375 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388418 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388454 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388482 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388516 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388565 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.388884 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.389849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.389801 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.390182 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.390159 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.390295 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.390275 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.390430 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.390405 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.393949 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.393924 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.394108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.394087 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.445130 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.445049 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.445130 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.445113 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44ht\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qrwgx\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.476120 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.476090 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:39.898806 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.898770 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx"] Nov 01 20:11:39.902236 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:11:39.902205 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c36d04f_fb39_4ac1_95f8_88a2767c5cfe.slice/crio-06dc357211d9092d37cac2b180e5ca917cdd3b369096ec8daa00dd592ff758bb WatchSource:0}: Error finding container 06dc357211d9092d37cac2b180e5ca917cdd3b369096ec8daa00dd592ff758bb: Status 404 returned error can't find the container with id 06dc357211d9092d37cac2b180e5ca917cdd3b369096ec8daa00dd592ff758bb Nov 01 20:11:39.904256 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:39.904240 5444 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:11:40.006588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:40.006552 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" event={"ID":"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe","Type":"ContainerStarted","Data":"06dc357211d9092d37cac2b180e5ca917cdd3b369096ec8daa00dd592ff758bb"} Nov 01 20:11:43.948420 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:43.948384 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:11:43.948735 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:43.948456 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:11:43.948735 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:43.948486 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:11:44.024886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:44.024858 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" event={"ID":"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe","Type":"ContainerStarted","Data":"8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872"} Nov 01 20:11:44.171478 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:44.171272 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podStartSLOduration=1.128333661 podStartE2EDuration="5.171254013s" totalImagesPullingTime="4.042920352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:44.170465034 +0000 UTC m=+505.115302903" watchObservedRunningTime="2026-11-01 20:11:44.171254013 +0000 UTC m=+505.116091881" Nov 01 20:11:44.477140 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:44.477097 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:45.478541 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:45.478504 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:45.478962 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:45.478573 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:46.477276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:46.477238 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:46.477456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:46.477292 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:47.477267 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:47.477224 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:47.477707 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:47.477289 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:48.476737 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:48.476699 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:48.476942 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:48.476760 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:49.476869 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:49.476817 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:49.477372 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:49.476881 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:50.477562 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:50.477523 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:50.477996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:50.477597 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:51.477086 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:51.477047 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qrwgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Nov 01 20:11:51.477292 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:51.477104 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Nov 01 20:11:53.128312 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:53.128275 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:53.128815 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:53.128755 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:53.129726 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:53.129703 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:11:54.299334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.299301 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk"] Nov 01 20:11:54.302506 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.302486 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.406502 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.406461 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk"] Nov 01 20:11:54.442053 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442021 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442061 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442144 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442215 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442253 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442280 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442317 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbz96\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442492 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442407 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.442492 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.442436 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543363 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543413 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543453 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543478 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543499 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543602 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543680 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbz96\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543725 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.543858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543752 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.544038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543886 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.544038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543902 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.544038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.543956 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.544176 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.544150 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.544496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.544478 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.545803 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.545784 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.546066 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.546045 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.600481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.600405 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.600682 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.600662 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbz96\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kc6jk\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:54.614072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:54.614047 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:55.036662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:55.036604 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk"] Nov 01 20:11:55.040396 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:11:55.040370 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cc12c9_6da4_43eb_9f66_01842db9605b.slice/crio-5fa18e7153bb1b82b948521d668d0902d0cfd0264b39402fbd7671d3189a627d WatchSource:0}: Error finding container 5fa18e7153bb1b82b948521d668d0902d0cfd0264b39402fbd7671d3189a627d: Status 404 returned error can't find the container with id 5fa18e7153bb1b82b948521d668d0902d0cfd0264b39402fbd7671d3189a627d Nov 01 20:11:55.042373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:55.042344 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:11:55.042438 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:55.042426 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:11:55.042473 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:55.042457 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:11:55.072861 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:55.072833 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" event={"ID":"73cc12c9-6da4-43eb-9f66-01842db9605b","Type":"ContainerStarted","Data":"5fa18e7153bb1b82b948521d668d0902d0cfd0264b39402fbd7671d3189a627d"} Nov 01 20:11:56.078799 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:56.078763 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" event={"ID":"73cc12c9-6da4-43eb-9f66-01842db9605b","Type":"ContainerStarted","Data":"c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568"} Nov 01 20:11:56.293511 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:56.293461 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podStartSLOduration=2.293441205 podStartE2EDuration="2.293441205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:56.244210026 +0000 UTC m=+517.189047896" watchObservedRunningTime="2026-11-01 20:11:56.293441205 +0000 UTC m=+517.238279075" Nov 01 20:11:56.614971 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:56.614933 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:11:56.616508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:56.616483 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:11:56.616626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:56.616539 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:11:57.615382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:57.615347 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:11:57.615778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:57.615404 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:11:58.614727 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:58.614685 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:11:58.614912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:58.614746 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:11:59.615081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:59.615044 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:11:59.615572 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:11:59.615106 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:12:00.035725 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:12:00.035601 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:12:00.614472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:00.614433 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:12:00.614688 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:00.614484 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:12:01.614618 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:01.614584 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:12:01.615174 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:01.614659 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:12:02.614957 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:02.614910 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 01 20:12:02.615508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:02.614976 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 01 20:12:04.242827 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:04.242792 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:12:04.243397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:04.243167 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:12:04.244117 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:04.244097 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:12:30.040008 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:12:30.039970 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:12:44.436609 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.436573 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt"] Nov 01 20:12:44.440824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.440797 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.492546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.492514 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-d204f4-serving-cert\"" Nov 01 20:12:44.540527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.540492 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.540702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.540532 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.540702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.540655 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.540702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.540702 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs56p\" (UniqueName: \"kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.542970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.542934 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt"] Nov 01 20:12:44.641101 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.641062 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.641101 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.641107 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.641361 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.641144 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.641361 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.641162 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs56p\" (UniqueName: \"kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.641361 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:12:44.641291 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-serving-cert: secret "maas-api-e2e-worker-w0-d204f4-serving-cert" not found Nov 01 20:12:44.641519 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:12:44.641368 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls podName:559d0cbf-8dd2-4ef7-b33b-056ae3643e12 nodeName:}" failed. No retries permitted until 2026-11-01 20:12:45.141345656 +0000 UTC m=+566.086183518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls") pod "maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" (UID: "559d0cbf-8dd2-4ef7-b33b-056ae3643e12") : secret "maas-api-e2e-worker-w0-d204f4-serving-cert" not found Nov 01 20:12:44.641805 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.641786 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.643616 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.643593 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:44.698407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:44.698334 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs56p\" (UniqueName: \"kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:45.145717 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:45.145682 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:45.148013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:45.147989 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:45.353219 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:45.353180 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:45.736053 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:45.735977 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt"] Nov 01 20:12:45.739699 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:12:45.739670 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559d0cbf_8dd2_4ef7_b33b_056ae3643e12.slice/crio-9e1fbfd8aff62fde6e3af2f5045c94ff826781d7cc4b6e62e82b227a0c5ed592 WatchSource:0}: Error finding container 9e1fbfd8aff62fde6e3af2f5045c94ff826781d7cc4b6e62e82b227a0c5ed592: Status 404 returned error can't find the container with id 9e1fbfd8aff62fde6e3af2f5045c94ff826781d7cc4b6e62e82b227a0c5ed592 Nov 01 20:12:46.286218 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:46.286179 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" event={"ID":"559d0cbf-8dd2-4ef7-b33b-056ae3643e12","Type":"ContainerStarted","Data":"9e1fbfd8aff62fde6e3af2f5045c94ff826781d7cc4b6e62e82b227a0c5ed592"} Nov 01 20:12:47.291813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:47.291770 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" event={"ID":"559d0cbf-8dd2-4ef7-b33b-056ae3643e12","Type":"ContainerStarted","Data":"8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955"} Nov 01 20:12:47.292329 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:47.291847 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:12:47.405458 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:47.405395 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" podStartSLOduration=2.529812761 podStartE2EDuration="3.405369023s" totalImagesPullingTime="875.556262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:47.404678628 +0000 UTC m=+568.349516494" watchObservedRunningTime="2026-11-01 20:12:47.405369023 +0000 UTC m=+568.350206888" Nov 01 20:12:53.302237 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:12:53.302208 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:13:00.035427 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:13:00.035385 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:13:09.921943 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:09.921909 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h"] Nov 01 20:13:09.925245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:09.925227 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.094455 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.094396 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.094748 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.094723 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.094933 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.094913 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.095079 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.095060 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.095208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.095190 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.096298 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.096274 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.096469 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.096451 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkxn\" (UniqueName: \"kubernetes.io/projected/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-kube-api-access-xjkxn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.096606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.096591 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.096746 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.096729 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.193411 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.193320 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h"] Nov 01 20:13:10.202130 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.202099 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.202379 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.202359 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.202524 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.202506 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.202672 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.202625 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkxn\" (UniqueName: \"kubernetes.io/projected/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-kube-api-access-xjkxn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.202819 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.202802 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.202938 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.202922 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.203098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.203082 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.203232 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.203217 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.203432 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.203416 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.213415 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.210210 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.213415 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.210561 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.225328 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.216998 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.225328 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.217048 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.225328 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.217359 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.225581 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.225446 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.226226 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.225852 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.299338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.299310 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkxn\" (UniqueName: \"kubernetes.io/projected/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-kube-api-access-xjkxn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.300115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.300087 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gg89h\" (UID: \"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:10.537333 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:10.537295 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:11.043951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.043912 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h"] Nov 01 20:13:11.048801 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:13:11.048020 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71d93d9_13cf_4a3d_afb4_3fdc6efc29a4.slice/crio-e0e9599dc85df052bd82ee5643feacac92a7a881c89ae80acbd8ad920fb63bf4 WatchSource:0}: Error finding container e0e9599dc85df052bd82ee5643feacac92a7a881c89ae80acbd8ad920fb63bf4: Status 404 returned error can't find the container with id e0e9599dc85df052bd82ee5643feacac92a7a881c89ae80acbd8ad920fb63bf4 Nov 01 20:13:11.050699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.050630 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:11.050809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.050751 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:11.050809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.050799 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:11.397984 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.397886 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" event={"ID":"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4","Type":"ContainerStarted","Data":"bd5524adc9a3c810e679210a3f7456bdd9545ea714f0edfd6905a9847aa0b72b"} Nov 01 20:13:11.398245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.398220 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" event={"ID":"d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4","Type":"ContainerStarted","Data":"e0e9599dc85df052bd82ee5643feacac92a7a881c89ae80acbd8ad920fb63bf4"} Nov 01 20:13:11.516203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.515968 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podStartSLOduration=2.515951216 podStartE2EDuration="2.515951216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:11.514765598 +0000 UTC m=+592.459603466" watchObservedRunningTime="2026-11-01 20:13:11.515951216 +0000 UTC m=+592.460789081" Nov 01 20:13:11.537855 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.537823 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:11.884666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.880779 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk"] Nov 01 20:13:11.885225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.885199 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.924329 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.924294 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.924507 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.924334 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.924507 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.924430 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.924507 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.924452 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhzn\" (UniqueName: \"kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.924507 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.924476 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.924749 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.924563 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:11.940071 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.939994 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"kube-root-ca.crt\"" Nov 01 20:13:11.940525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.940479 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"openshift-service-ca.crt\"" Nov 01 20:13:11.941727 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.941692 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk"] Nov 01 20:13:11.981094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.981032 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"simulator-w3-1b1134-kserve-self-signed-certs\"" Nov 01 20:13:11.989249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:11.989210 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"default-dockercfg-t4w2r\"" Nov 01 20:13:12.026097 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.026056 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.026364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.026338 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.026567 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.026544 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.026778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.026759 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.026943 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.026922 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhzn\" (UniqueName: \"kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.027123 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.026777 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.027705 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.027088 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.027883 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.027865 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.028363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.028260 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.032102 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.032078 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.034070 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.034045 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.085759 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.085724 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhzn\" (UniqueName: \"kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn\") pod \"simulator-w3-1b1134-kserve-644dc6d568-7kcdk\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.199964 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.199857 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:12.539336 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.539303 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:12.539523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.539364 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:12.598695 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:12.598615 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk"] Nov 01 20:13:12.602665 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:13:12.602622 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb17a633_8bf3_4d93_9ee2_ac49378aa623.slice/crio-1bd032dcd7a40ccd5a72072b6c2e7dacd4a71550856114b8bf85e841a1501824 WatchSource:0}: Error finding container 1bd032dcd7a40ccd5a72072b6c2e7dacd4a71550856114b8bf85e841a1501824: Status 404 returned error can't find the container with id 1bd032dcd7a40ccd5a72072b6c2e7dacd4a71550856114b8bf85e841a1501824 Nov 01 20:13:13.410576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:13.410532 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" event={"ID":"db17a633-8bf3-4d93-9ee2-ac49378aa623","Type":"ContainerStarted","Data":"1bd032dcd7a40ccd5a72072b6c2e7dacd4a71550856114b8bf85e841a1501824"} Nov 01 20:13:13.543472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:13.542755 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:13.543472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:13.542853 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:14.537727 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:14.537691 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:14.538137 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:14.537748 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:15.421702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:15.421666 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" event={"ID":"db17a633-8bf3-4d93-9ee2-ac49378aa623","Type":"ContainerStarted","Data":"2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c"} Nov 01 20:13:15.537128 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:15.537054 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" podStartSLOduration=2.583265486 podStartE2EDuration="4.537033586s" totalImagesPullingTime="1.9537681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:15.536281379 +0000 UTC m=+596.481119248" watchObservedRunningTime="2026-11-01 20:13:15.537033586 +0000 UTC m=+596.481871453" Nov 01 20:13:15.538306 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:15.538273 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:15.538801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:15.538336 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:16.200432 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:16.200380 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:16.538413 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:16.538378 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:16.538839 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:16.538436 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:17.538167 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:17.538127 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:17.538333 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:17.538201 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:18.538243 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:18.538208 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Nov 01 20:13:18.538779 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:18.538269 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Nov 01 20:13:20.533400 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:20.533367 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:20.533896 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:20.533733 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:20.534540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:20.534521 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" Nov 01 20:13:22.200595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:22.200558 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:22.212934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:22.212904 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:22.468605 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:22.468528 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:13:30.034631 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:13:30.034593 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:13:36.248771 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.248736 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp"] Nov 01 20:13:36.255720 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.255691 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.358301 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.358263 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"kube-root-ca.crt\"" Nov 01 20:13:36.358301 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.358290 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"openshift-service-ca.crt\"" Nov 01 20:13:36.358535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.358423 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"default-dockercfg-qdjvd\"" Nov 01 20:13:36.358535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.358474 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"simulator-w0-d204f4-kserve-self-signed-certs\"" Nov 01 20:13:36.359446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.359424 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp"] Nov 01 20:13:36.404848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.404813 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.404848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.404848 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25rc\" (UniqueName: \"kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.405108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.404875 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.405108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.405023 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.405108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.405070 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.405218 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.405170 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506035 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506079 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25rc\" (UniqueName: \"kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506109 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506159 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506183 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506232 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506558 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506679 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506651 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.506841 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.506820 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.508445 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.508419 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.508710 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.508691 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.563198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.563150 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25rc\" (UniqueName: \"kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc\") pod \"simulator-w0-d204f4-kserve-69794c99f5-5kbrp\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.568824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.568785 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:36.944447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:36.944411 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp"] Nov 01 20:13:36.948071 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:13:36.948046 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53fe30f6_048b_4ddb_8c8e_33c4ca486e36.slice/crio-e31961455f753ee5b87753868089041dc81c19067b6ba69f9138c3261b7a6c85 WatchSource:0}: Error finding container e31961455f753ee5b87753868089041dc81c19067b6ba69f9138c3261b7a6c85: Status 404 returned error can't find the container with id e31961455f753ee5b87753868089041dc81c19067b6ba69f9138c3261b7a6c85 Nov 01 20:13:37.521057 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:37.521024 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" event={"ID":"53fe30f6-048b-4ddb-8c8e-33c4ca486e36","Type":"ContainerStarted","Data":"23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6"} Nov 01 20:13:37.521477 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:37.521065 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" event={"ID":"53fe30f6-048b-4ddb-8c8e-33c4ca486e36","Type":"ContainerStarted","Data":"e31961455f753ee5b87753868089041dc81c19067b6ba69f9138c3261b7a6c85"} Nov 01 20:13:37.569847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:37.569807 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:37.637396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:37.637336 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" podStartSLOduration=1.6373192460000001 podStartE2EDuration="1.637319246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:37.636619279 +0000 UTC m=+618.581457147" watchObservedRunningTime="2026-11-01 20:13:37.637319246 +0000 UTC m=+618.582157113" Nov 01 20:13:45.411341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.411303 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz"] Nov 01 20:13:45.414366 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.414349 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.516980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.516945 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"premium-w3-1b1134-kserve-self-signed-certs\"" Nov 01 20:13:45.517900 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.517879 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz"] Nov 01 20:13:45.602624 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.602588 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.602815 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.602680 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgl8\" (UniqueName: \"kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.602815 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.602782 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.602893 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.602823 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.602893 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.602877 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.602957 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.602903 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704152 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgl8\" (UniqueName: \"kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704215 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704237 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704261 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704287 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704323 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704732 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704807 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.704958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.704939 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.706604 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.706579 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.707119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.707102 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:45.761047 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:45.761013 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgl8\" (UniqueName: \"kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8\") pod \"premium-w3-1b1134-kserve-746746777c-6vfxz\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:46.026009 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.025974 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:46.398531 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.398456 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz"] Nov 01 20:13:46.401740 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:13:46.401707 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4439e41_5d46_4b30_8b4e_d5d83f3e9520.slice/crio-6d6b1aa198a119257f2bf874c8c7286676a82b636d14a12544ca1fcfd554b1d5 WatchSource:0}: Error finding container 6d6b1aa198a119257f2bf874c8c7286676a82b636d14a12544ca1fcfd554b1d5: Status 404 returned error can't find the container with id 6d6b1aa198a119257f2bf874c8c7286676a82b636d14a12544ca1fcfd554b1d5 Nov 01 20:13:46.568950 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.568910 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:46.570198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.570165 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" event={"ID":"f4439e41-5d46-4b30-8b4e-d5d83f3e9520","Type":"ContainerStarted","Data":"d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5"} Nov 01 20:13:46.570338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.570214 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" event={"ID":"f4439e41-5d46-4b30-8b4e-d5d83f3e9520","Type":"ContainerStarted","Data":"6d6b1aa198a119257f2bf874c8c7286676a82b636d14a12544ca1fcfd554b1d5"} Nov 01 20:13:46.581404 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.581383 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:46.683249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:46.683146 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" podStartSLOduration=1.683128826 podStartE2EDuration="1.683128826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:46.682248032 +0000 UTC m=+627.627085899" watchObservedRunningTime="2026-11-01 20:13:46.683128826 +0000 UTC m=+627.627966695" Nov 01 20:13:47.026767 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:47.026736 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:47.581261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:47.581231 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:13:56.026891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.026799 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:56.039304 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.039277 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:56.359866 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.359777 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj"] Nov 01 20:13:56.363259 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.363234 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.388604 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.388577 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5n2g\" (UniqueName: \"kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.388751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.388616 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.388751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.388649 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.388751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.388710 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.415360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.415327 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-df41b8-serving-cert\"" Nov 01 20:13:56.423398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.423369 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj"] Nov 01 20:13:56.490159 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.490123 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5n2g\" (UniqueName: \"kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.490344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.490185 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.490344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.490216 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.490344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.490275 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.490524 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:13:56.490415 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-serving-cert: secret "maas-api-e2e-shared-a-w4-df41b8-serving-cert" not found Nov 01 20:13:56.490524 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:13:56.490487 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls podName:6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555 nodeName:}" failed. No retries permitted until 2026-11-01 20:13:56.990465913 +0000 UTC m=+637.935303776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls") pod "maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" (UID: "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555") : secret "maas-api-e2e-shared-a-w4-df41b8-serving-cert" not found Nov 01 20:13:56.491026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.491005 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.492815 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.492792 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.549438 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.549400 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5n2g\" (UniqueName: \"kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.636791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.636712 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:13:56.995755 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.995680 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:56.997878 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:56.997850 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:57.275823 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:57.275788 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:57.646347 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:57.646268 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj"] Nov 01 20:13:57.649969 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:13:57.649941 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecdc2db_0bfa_4b3f_8d4d_af9cae29f555.slice/crio-80be74b15a3b34fa2d176f31048f1948ec1e4921566a1103a92d3c6decb2e10a WatchSource:0}: Error finding container 80be74b15a3b34fa2d176f31048f1948ec1e4921566a1103a92d3c6decb2e10a: Status 404 returned error can't find the container with id 80be74b15a3b34fa2d176f31048f1948ec1e4921566a1103a92d3c6decb2e10a Nov 01 20:13:58.631130 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:58.631081 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" event={"ID":"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555","Type":"ContainerStarted","Data":"1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6"} Nov 01 20:13:58.631499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:58.631153 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" event={"ID":"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555","Type":"ContainerStarted","Data":"80be74b15a3b34fa2d176f31048f1948ec1e4921566a1103a92d3c6decb2e10a"} Nov 01 20:13:58.631499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:58.631187 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:13:58.806314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:13:58.806259 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" podStartSLOduration=1.955187775 podStartE2EDuration="2.806242342s" totalImagesPullingTime="851.054567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:58.80501805 +0000 UTC m=+639.749855919" watchObservedRunningTime="2026-11-01 20:13:58.806242342 +0000 UTC m=+639.751080207" Nov 01 20:14:00.036119 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:14:00.036082 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:14:04.641740 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:04.641706 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:14:15.060389 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.060352 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8"] Nov 01 20:14:15.064887 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.064857 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.122532 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.122498 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8"] Nov 01 20:14:15.163699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.163470 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.163699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.163538 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.163699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.163593 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.163699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.163625 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2w6\" (UniqueName: \"kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.163699 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.163703 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.164117 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.163802 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.168282 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.168248 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d204f4\"/\"premium-w0-d204f4-kserve-self-signed-certs\"" Nov 01 20:14:15.265236 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265203 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265236 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265246 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265380 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265412 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2w6\" (UniqueName: \"kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265555 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265464 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265555 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265498 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265714 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265688 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265771 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265726 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.265837 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.265793 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.267785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.267758 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.267895 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.267882 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.329352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.329273 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2w6\" (UniqueName: \"kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6\") pod \"premium-w0-d204f4-kserve-6b9c6dd974-bv9m8\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.379038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.379001 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:15.782104 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:15.782071 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8"] Nov 01 20:14:15.786062 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:14:15.786028 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70018ba0_cddf_45fd_90c2_843278f3bfd7.slice/crio-a34128915bde976008c2a24f4820a8a3f65a7f0a04e23e4d085d7a969e1453b6 WatchSource:0}: Error finding container a34128915bde976008c2a24f4820a8a3f65a7f0a04e23e4d085d7a969e1453b6: Status 404 returned error can't find the container with id a34128915bde976008c2a24f4820a8a3f65a7f0a04e23e4d085d7a969e1453b6 Nov 01 20:14:16.712158 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:16.712120 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" event={"ID":"70018ba0-cddf-45fd-90c2-843278f3bfd7","Type":"ContainerStarted","Data":"7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766"} Nov 01 20:14:16.712716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:16.712177 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" event={"ID":"70018ba0-cddf-45fd-90c2-843278f3bfd7","Type":"ContainerStarted","Data":"a34128915bde976008c2a24f4820a8a3f65a7f0a04e23e4d085d7a969e1453b6"} Nov 01 20:14:16.826076 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:16.825936 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" podStartSLOduration=1.825921782 podStartE2EDuration="1.825921782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:16.825104221 +0000 UTC m=+657.769942100" watchObservedRunningTime="2026-11-01 20:14:16.825921782 +0000 UTC m=+657.770759648" Nov 01 20:14:17.379608 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:17.379575 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:25.379817 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:25.379776 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:25.392227 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:25.392197 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:25.762957 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:14:25.762923 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:14:30.034863 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:14:30.034827 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:15:00.034668 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:00.034605 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:15:00.179208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.179168 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6"] Nov 01 20:15:00.182550 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.182522 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.240541 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.240505 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:15:00.241563 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.241536 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkrf\" (UniqueName: \"kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf\") pod \"maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.241744 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.241671 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.293903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.293800 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6"] Nov 01 20:15:00.342438 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.342404 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.342618 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.342467 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkrf\" (UniqueName: \"kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf\") pod \"maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.343119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.343098 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.399174 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.399136 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkrf\" (UniqueName: \"kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf\") pod \"maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.494798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.494763 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:00.865602 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.865529 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6"] Nov 01 20:15:00.869247 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:15:00.869212 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885ed24b_ea25_4326_929d_e8ec6290b7dc.slice/crio-a113c12765f874207a593b7c2cd5a6925762571994b96d90afaef73834e66111 WatchSource:0}: Error finding container a113c12765f874207a593b7c2cd5a6925762571994b96d90afaef73834e66111: Status 404 returned error can't find the container with id a113c12765f874207a593b7c2cd5a6925762571994b96d90afaef73834e66111 Nov 01 20:15:00.911787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:00.911754 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" event={"ID":"885ed24b-ea25-4326-929d-e8ec6290b7dc","Type":"ContainerStarted","Data":"a113c12765f874207a593b7c2cd5a6925762571994b96d90afaef73834e66111"} Nov 01 20:15:03.470397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.470363 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7"] Nov 01 20:15:03.474465 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.474440 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.574886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.574846 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.574886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.574884 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.575120 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.574955 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.575120 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.574977 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.575120 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.575004 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.575120 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.575022 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jl5\" (UniqueName: \"kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.575846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.575819 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1b1134\"/\"e2e-conflict-a-b637b059-kserve-self-signed-certs\"" Nov 01 20:15:03.576944 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.576924 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7"] Nov 01 20:15:03.676368 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676321 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676368 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676387 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676430 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676453 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jl5\" (UniqueName: \"kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676557 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676580 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676839 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676925 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676886 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.676996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.676975 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.678963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.678936 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.679396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.679372 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.736819 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.736736 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jl5\" (UniqueName: \"kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5\") pod \"e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:03.786791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:03.786522 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:04.212779 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:04.212743 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7"] Nov 01 20:15:04.217251 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:15:04.217210 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f5fec9_f6c2_46bc_b537_bf0ac92bc633.slice/crio-730c54f6f9960c135de52e468a87ff3ab8a0caf37049f5a1383fd9d4b639eb0c WatchSource:0}: Error finding container 730c54f6f9960c135de52e468a87ff3ab8a0caf37049f5a1383fd9d4b639eb0c: Status 404 returned error can't find the container with id 730c54f6f9960c135de52e468a87ff3ab8a0caf37049f5a1383fd9d4b639eb0c Nov 01 20:15:04.937220 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:04.936278 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" event={"ID":"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633","Type":"ContainerStarted","Data":"1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307"} Nov 01 20:15:04.937220 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:04.936333 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" event={"ID":"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633","Type":"ContainerStarted","Data":"730c54f6f9960c135de52e468a87ff3ab8a0caf37049f5a1383fd9d4b639eb0c"} Nov 01 20:15:05.052091 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:05.052026 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" podStartSLOduration=2.052010818 podStartE2EDuration="2.052010818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:05.050796135 +0000 UTC m=+705.995634005" watchObservedRunningTime="2026-11-01 20:15:05.052010818 +0000 UTC m=+705.996848683" Nov 01 20:15:05.787010 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:05.786968 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:05.942383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:05.942288 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" event={"ID":"885ed24b-ea25-4326-929d-e8ec6290b7dc","Type":"ContainerStarted","Data":"61a5ea114ba599156990b4a3d476a7d5ad898ad4c05d56f958a495a7c07f12e4"} Nov 01 20:15:06.057052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:06.056998 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" podStartSLOduration=1.681983433 podStartE2EDuration="6.056984526s" totalImagesPullingTime="4.375001093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:06.05574596 +0000 UTC m=+707.000583827" watchObservedRunningTime="2026-11-01 20:15:06.056984526 +0000 UTC m=+707.001822393" Nov 01 20:15:06.948337 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:06.948298 5444 generic.go:356] "Generic (PLEG): container finished" podID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerID="61a5ea114ba599156990b4a3d476a7d5ad898ad4c05d56f958a495a7c07f12e4" exitCode=60 Nov 01 20:15:06.948753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:06.948366 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" event={"ID":"885ed24b-ea25-4326-929d-e8ec6290b7dc","Type":"ContainerDied","Data":"61a5ea114ba599156990b4a3d476a7d5ad898ad4c05d56f958a495a7c07f12e4"} Nov 01 20:15:06.948753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:06.948650 5444 scope.go:122] "RemoveContainer" containerID="61a5ea114ba599156990b4a3d476a7d5ad898ad4c05d56f958a495a7c07f12e4" Nov 01 20:15:07.954538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:07.954498 5444 generic.go:356] "Generic (PLEG): container finished" podID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerID="0aeed2e0b210a3197fd26809385e2447a7f3235fbf3fc7aa2a700dbaf0ba49b4" exitCode=60 Nov 01 20:15:07.954975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:07.954587 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" event={"ID":"885ed24b-ea25-4326-929d-e8ec6290b7dc","Type":"ContainerDied","Data":"0aeed2e0b210a3197fd26809385e2447a7f3235fbf3fc7aa2a700dbaf0ba49b4"} Nov 01 20:15:07.954975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:07.954690 5444 scope.go:122] "RemoveContainer" containerID="61a5ea114ba599156990b4a3d476a7d5ad898ad4c05d56f958a495a7c07f12e4" Nov 01 20:15:07.954975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:07.954818 5444 scope.go:122] "RemoveContainer" containerID="0aeed2e0b210a3197fd26809385e2447a7f3235fbf3fc7aa2a700dbaf0ba49b4" Nov 01 20:15:07.955168 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:07.955075 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6_odh-ai-gateway-infra(885ed24b-ea25-4326-929d-e8ec6290b7dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" Nov 01 20:15:08.961494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:08.961467 5444 scope.go:122] "RemoveContainer" containerID="0aeed2e0b210a3197fd26809385e2447a7f3235fbf3fc7aa2a700dbaf0ba49b4" Nov 01 20:15:08.961907 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:08.961716 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6_odh-ai-gateway-infra(885ed24b-ea25-4326-929d-e8ec6290b7dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" Nov 01 20:15:10.042655 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.042603 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5"] Nov 01 20:15:10.047138 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.047119 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.098939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.098907 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2cc24d-serving-cert\"" Nov 01 20:15:10.141510 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.141475 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.141690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.141522 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.141690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.141542 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.141690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.141670 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrn4\" (UniqueName: \"kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.150266 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.150237 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5"] Nov 01 20:15:10.242505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.242470 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.242734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.242540 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.242734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.242569 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.242734 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:10.242664 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-serving-cert: secret "maas-api-e2e-shared-b-w4-2cc24d-serving-cert" not found Nov 01 20:15:10.242912 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:10.242746 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls podName:32ac1051-c4df-4936-badc-06620530e792 nodeName:}" failed. No retries permitted until 2026-11-01 20:15:10.742723583 +0000 UTC m=+711.687561444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls") pod "maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" (UID: "32ac1051-c4df-4936-badc-06620530e792") : secret "maas-api-e2e-shared-b-w4-2cc24d-serving-cert" not found Nov 01 20:15:10.243070 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.243040 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrn4\" (UniqueName: \"kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.243286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.243257 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.245121 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.245099 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.307745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.307632 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrn4\" (UniqueName: \"kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.748583 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.748496 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.751096 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.751067 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:10.959193 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:10.959153 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:11.331449 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:11.331374 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5"] Nov 01 20:15:11.334429 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:15:11.334397 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ac1051_c4df_4936_badc_06620530e792.slice/crio-b031adfc4134527e020d1c8be6c439a4d505cd4726c5e91defe557f7609127ad WatchSource:0}: Error finding container b031adfc4134527e020d1c8be6c439a4d505cd4726c5e91defe557f7609127ad: Status 404 returned error can't find the container with id b031adfc4134527e020d1c8be6c439a4d505cd4726c5e91defe557f7609127ad Nov 01 20:15:11.974761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:11.974721 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" event={"ID":"32ac1051-c4df-4936-badc-06620530e792","Type":"ContainerStarted","Data":"b031adfc4134527e020d1c8be6c439a4d505cd4726c5e91defe557f7609127ad"} Nov 01 20:15:12.980958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:12.980924 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" event={"ID":"32ac1051-c4df-4936-badc-06620530e792","Type":"ContainerStarted","Data":"0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276"} Nov 01 20:15:12.981357 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:12.981065 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:13.094600 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:13.094541 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" podStartSLOduration=2.15306847 podStartE2EDuration="3.094525673s" totalImagesPullingTime="941.457203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:13.094200634 +0000 UTC m=+714.039038502" watchObservedRunningTime="2026-11-01 20:15:13.094525673 +0000 UTC m=+714.039363540" Nov 01 20:15:13.787102 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:13.787065 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:13.799790 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:13.799758 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:13.996585 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:13.996557 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:15:18.991154 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:18.991125 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:15:22.152317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:22.152278 5444 scope.go:122] "RemoveContainer" containerID="0aeed2e0b210a3197fd26809385e2447a7f3235fbf3fc7aa2a700dbaf0ba49b4" Nov 01 20:15:23.029423 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:23.029390 5444 generic.go:356] "Generic (PLEG): container finished" podID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerID="7676d93bfb089e9e0f1b769ea9f516ddbbc651ffc31afc842973878785891f13" exitCode=60 Nov 01 20:15:23.029602 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:23.029467 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" event={"ID":"885ed24b-ea25-4326-929d-e8ec6290b7dc","Type":"ContainerDied","Data":"7676d93bfb089e9e0f1b769ea9f516ddbbc651ffc31afc842973878785891f13"} Nov 01 20:15:23.029602 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:23.029540 5444 scope.go:122] "RemoveContainer" containerID="0aeed2e0b210a3197fd26809385e2447a7f3235fbf3fc7aa2a700dbaf0ba49b4" Nov 01 20:15:23.029901 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:23.029877 5444 scope.go:122] "RemoveContainer" containerID="7676d93bfb089e9e0f1b769ea9f516ddbbc651ffc31afc842973878785891f13" Nov 01 20:15:23.030143 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:23.030119 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6_odh-ai-gateway-infra(885ed24b-ea25-4326-929d-e8ec6290b7dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" Nov 01 20:15:24.150277 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.150244 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6"] Nov 01 20:15:24.278189 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.278160 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:24.380951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.380918 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf\" (UniqueName: \"kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf\") pod \"885ed24b-ea25-4326-929d-e8ec6290b7dc\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " Nov 01 20:15:24.381165 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.380974 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca\") pod \"885ed24b-ea25-4326-929d-e8ec6290b7dc\" (UID: \"885ed24b-ea25-4326-929d-e8ec6290b7dc\") " Nov 01 20:15:24.381403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.381372 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca" pod "885ed24b-ea25-4326-929d-e8ec6290b7dc" (UID: "885ed24b-ea25-4326-929d-e8ec6290b7dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:24.383205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.383178 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf" pod "885ed24b-ea25-4326-929d-e8ec6290b7dc" (UID: "885ed24b-ea25-4326-929d-e8ec6290b7dc"). InnerVolumeSpecName "kube-api-access-9mkrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:24.482235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.482150 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mkrf\" (UniqueName: \"kubernetes.io/projected/885ed24b-ea25-4326-929d-e8ec6290b7dc-kube-api-access-9mkrf\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:24.482235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:24.482180 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/885ed24b-ea25-4326-929d-e8ec6290b7dc-maas-api-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:15:25.041471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:25.041441 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" Nov 01 20:15:25.041471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:25.041466 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6" event={"ID":"885ed24b-ea25-4326-929d-e8ec6290b7dc","Type":"ContainerDied","Data":"a113c12765f874207a593b7c2cd5a6925762571994b96d90afaef73834e66111"} Nov 01 20:15:25.041751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:25.041527 5444 scope.go:122] "RemoveContainer" containerID="7676d93bfb089e9e0f1b769ea9f516ddbbc651ffc31afc842973878785891f13" Nov 01 20:15:25.204566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:25.204532 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6"] Nov 01 20:15:25.253575 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:25.253545 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-381df3-29892735-ldvt6"] Nov 01 20:15:27.158003 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:27.157968 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" path="/var/lib/kubelet/pods/885ed24b-ea25-4326-929d-e8ec6290b7dc/volumes" Nov 01 20:15:30.034353 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:30.034315 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:15:34.062497 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062462 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz"] Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062840 5444 cpu_manager.go:414] "containerMap: removing container" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062852 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062865 5444 cpu_manager.go:414] "containerMap: removing container" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062870 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062887 5444 cpu_manager.go:414] "containerMap: removing container" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062893 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062946 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.062970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.062957 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:34.067439 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.067416 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.076122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.075718 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9cb\" (UniqueName: \"kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.076122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.075770 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.076122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.075796 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.076122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.075916 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.122302 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.122266 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ad7edb-serving-cert\"" Nov 01 20:15:34.169761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.169729 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz"] Nov 01 20:15:34.177071 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.177035 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9cb\" (UniqueName: \"kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.177214 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.177078 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.177214 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.177121 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.177214 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:34.177163 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-serving-cert: secret "maas-api-e2e-inf-a-ad7edb-serving-cert" not found Nov 01 20:15:34.177214 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.177173 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.177405 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:15:34.177223 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls podName:700a1795-58fc-4939-aba9-2f7c070d136b nodeName:}" failed. No retries permitted until 2026-11-01 20:15:34.677202331 +0000 UTC m=+735.622040197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls") pod "maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" (UID: "700a1795-58fc-4939-aba9-2f7c070d136b") : secret "maas-api-e2e-inf-a-ad7edb-serving-cert" not found Nov 01 20:15:34.178707 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.178677 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.181496 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.181469 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.235701 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.235667 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9cb\" (UniqueName: \"kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.683430 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.683393 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.685776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.685752 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:34.983190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:34.983089 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:35.383468 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:35.383397 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz"] Nov 01 20:15:35.386427 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:15:35.386396 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700a1795_58fc_4939_aba9_2f7c070d136b.slice/crio-06f51cf6ee8f2fcc2868e5a1417d2bc2490c3206b1b906c0165159b51f6d8bf1 WatchSource:0}: Error finding container 06f51cf6ee8f2fcc2868e5a1417d2bc2490c3206b1b906c0165159b51f6d8bf1: Status 404 returned error can't find the container with id 06f51cf6ee8f2fcc2868e5a1417d2bc2490c3206b1b906c0165159b51f6d8bf1 Nov 01 20:15:36.094144 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:36.094110 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" event={"ID":"700a1795-58fc-4939-aba9-2f7c070d136b","Type":"ContainerStarted","Data":"06f51cf6ee8f2fcc2868e5a1417d2bc2490c3206b1b906c0165159b51f6d8bf1"} Nov 01 20:15:37.100321 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:37.100281 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" event={"ID":"700a1795-58fc-4939-aba9-2f7c070d136b","Type":"ContainerStarted","Data":"78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733"} Nov 01 20:15:37.100818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:37.100374 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:37.219291 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:37.219232 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" podStartSLOduration=2.34357669 podStartE2EDuration="3.219217195s" totalImagesPullingTime="875.640505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:37.217566219 +0000 UTC m=+738.162404087" watchObservedRunningTime="2026-11-01 20:15:37.219217195 +0000 UTC m=+738.164055063" Nov 01 20:15:43.111328 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:43.111293 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:15:44.959307 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:44.959270 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf"] Nov 01 20:15:44.959760 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:44.959745 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="885ed24b-ea25-4326-929d-e8ec6290b7dc" containerName="cleanup" Nov 01 20:15:44.966494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:44.966472 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.018540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.018502 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1b1134-openshift-default-dockercfg-pvzt4\"" Nov 01 20:15:45.069529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.069491 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf"] Nov 01 20:15:45.079840 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.079780 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.079840 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.079830 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.079862 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.079891 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.079920 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhfc\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.079989 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.080167 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.080206 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.080355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.080227 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181323 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181289 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181323 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181322 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181345 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhfc\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181418 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181511 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181563 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181595 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181692 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181722 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181764 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.181870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181826 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.182035 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.181981 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.182035 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.182021 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.182396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.182372 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.184125 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.184106 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.184376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.184354 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.237338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.237261 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.237467 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.237372 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhfc\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.279337 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.279297 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:45.705888 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.705712 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf"] Nov 01 20:15:45.708755 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:15:45.708722 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15b1f8b_b1c4_48e9_bd69_96096a751ac6.slice/crio-a7c69132547f922e4c9f85890d6fc6de589ae69b89cb2fcf3f42187ce40526ef WatchSource:0}: Error finding container a7c69132547f922e4c9f85890d6fc6de589ae69b89cb2fcf3f42187ce40526ef: Status 404 returned error can't find the container with id a7c69132547f922e4c9f85890d6fc6de589ae69b89cb2fcf3f42187ce40526ef Nov 01 20:15:45.711106 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.711074 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:15:45.711218 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.711155 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:15:45.711218 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:45.711205 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:15:46.144145 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:46.144105 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" event={"ID":"a15b1f8b-b1c4-48e9-bd69-96096a751ac6","Type":"ContainerStarted","Data":"4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97"} Nov 01 20:15:46.144573 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:46.144169 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" event={"ID":"a15b1f8b-b1c4-48e9-bd69-96096a751ac6","Type":"ContainerStarted","Data":"a7c69132547f922e4c9f85890d6fc6de589ae69b89cb2fcf3f42187ce40526ef"} Nov 01 20:15:46.280016 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:46.279970 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:46.281837 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:46.281800 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:46.282084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:46.281864 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:47.279966 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:47.279932 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:47.280420 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:47.279988 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:48.279994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:48.279954 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:48.280410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:48.280009 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:49.280268 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:49.280229 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:49.280734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:49.280286 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:50.279618 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:50.279585 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:50.279815 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:50.279665 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:51.279837 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:51.279802 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:51.280226 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:51.279850 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:52.280160 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:52.280112 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 01 20:15:52.280687 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:52.280188 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 01 20:15:53.634771 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:53.634739 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:53.635274 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:53.635072 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:53.636044 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:53.636018 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:15:53.798992 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:15:53.798937 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podStartSLOduration=9.798920684 podStartE2EDuration="9.798920684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:46.261901872 +0000 UTC m=+747.206739746" watchObservedRunningTime="2026-11-01 20:15:53.798920684 +0000 UTC m=+754.743758553" Nov 01 20:16:00.034840 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:16:00.034797 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:16:14.973383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:14.973349 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw"] Nov 01 20:16:14.976963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:14.976947 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.029939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.029913 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\"" Nov 01 20:16:15.081556 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.081521 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw"] Nov 01 20:16:15.100274 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100237 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100274 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100280 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100490 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100307 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100490 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100400 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100490 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100428 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100490 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100446 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100580 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtdc\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100614 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.100658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.100651 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201256 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201293 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201492 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201321 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201545 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201516 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201599 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201574 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201682 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201605 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201757 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201736 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtdc\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201781 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201815 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201856 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201742 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201922 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.201958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.201942 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.202354 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.202222 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.202532 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.202507 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.203864 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.203842 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.203982 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.203965 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.259441 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.259406 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.259713 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.259689 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtdc\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.291551 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.291523 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:15.724568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.724487 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw"] Nov 01 20:16:15.727954 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:16:15.727927 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1795b94e_b55b_4d4c_96ef_6a20d0d6b5b9.slice/crio-7c427ea6b1fb75f3776714bc5d04aed94677c01a968ac0e5bc8a031b68603f55 WatchSource:0}: Error finding container 7c427ea6b1fb75f3776714bc5d04aed94677c01a968ac0e5bc8a031b68603f55: Status 404 returned error can't find the container with id 7c427ea6b1fb75f3776714bc5d04aed94677c01a968ac0e5bc8a031b68603f55 Nov 01 20:16:15.730249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.730215 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:16:15.730314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.730284 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:16:15.730364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:15.730315 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:16:16.314976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:16.314939 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" event={"ID":"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9","Type":"ContainerStarted","Data":"bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245"} Nov 01 20:16:16.314976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:16.314993 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" event={"ID":"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9","Type":"ContainerStarted","Data":"7c427ea6b1fb75f3776714bc5d04aed94677c01a968ac0e5bc8a031b68603f55"} Nov 01 20:16:16.434870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:16.434805 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podStartSLOduration=2.4347863370000002 podStartE2EDuration="2.434786337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:16.43336628 +0000 UTC m=+777.378204148" watchObservedRunningTime="2026-11-01 20:16:16.434786337 +0000 UTC m=+777.379624212" Nov 01 20:16:17.292212 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:17.292169 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:17.293707 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:17.293679 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:16:17.293836 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:17.293731 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:16:18.292425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:18.292391 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:16:18.292857 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:18.292444 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:16:19.292435 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:19.292402 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:16:19.292819 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:19.292467 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:16:20.292373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:20.292343 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:16:20.292555 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:20.292392 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:16:21.292210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:21.292161 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:16:21.292386 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:21.292236 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:16:22.292244 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:22.292201 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:16:22.292667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:22.292272 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:16:23.648912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:23.648875 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:23.649373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:23.649188 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:23.650059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:23.650043 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:16:30.034819 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:16:30.034779 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:16:38.992368 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:38.992328 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7"] Nov 01 20:16:54.012427 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.012382 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" podUID="e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" containerName="main" containerID="cri-o://1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307" gracePeriod=45 Nov 01 20:16:54.259016 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.258987 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:16:54.291917 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.291838 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home\") pod \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " Nov 01 20:16:54.292072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.291929 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache\") pod \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " Nov 01 20:16:54.292072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.291977 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs\") pod \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " Nov 01 20:16:54.292072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292006 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm\") pod \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " Nov 01 20:16:54.292072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292045 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5\" (UniqueName: \"kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5\") pod \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " Nov 01 20:16:54.292300 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292085 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir\") pod \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\" (UID: \"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633\") " Nov 01 20:16:54.292300 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292092 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home" pod "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" (UID: "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:54.292300 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292227 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache" pod "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" (UID: "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:54.292441 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292367 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir" pod "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" (UID: "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:54.292441 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292435 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:54.292557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292446 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:54.292557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.292455 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:54.294283 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.294251 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm" pod "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" (UID: "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:16:54.294390 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.294284 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs" pod "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" (UID: "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:16:54.294432 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.294374 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5" pod "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" (UID: "e3f5fec9-f6c2-46bc-b537-bf0ac92bc633"). InnerVolumeSpecName "kube-api-access-w6jl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:16:54.393317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.393279 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:54.393317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.393313 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:54.393317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.393328 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6jl5\" (UniqueName: \"kubernetes.io/projected/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633-kube-api-access-w6jl5\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:16:54.548080 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.547994 5444 generic.go:356] "Generic (PLEG): container finished" podID="e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" containerID="1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307" exitCode=0 Nov 01 20:16:54.548080 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.548039 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" Nov 01 20:16:54.548288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.548042 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" event={"ID":"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633","Type":"ContainerDied","Data":"1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307"} Nov 01 20:16:54.548288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.548161 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7" event={"ID":"e3f5fec9-f6c2-46bc-b537-bf0ac92bc633","Type":"ContainerDied","Data":"730c54f6f9960c135de52e468a87ff3ab8a0caf37049f5a1383fd9d4b639eb0c"} Nov 01 20:16:54.548288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.548195 5444 scope.go:122] "RemoveContainer" containerID="1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307" Nov 01 20:16:54.563783 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.563760 5444 scope.go:122] "RemoveContainer" containerID="1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307" Nov 01 20:16:54.564022 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:16:54.564002 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307\": container with ID starting with 1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307 not found: ID does not exist" containerID="1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307" Nov 01 20:16:54.564067 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.564032 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307"} err="failed to get container status \"1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307\": rpc error: code = NotFound desc = could not find container \"1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307\": container with ID starting with 1f5299f7b09b742a66a30c0d263d712aac0e42ccfda61c2d84d1ce1777541307 not found: ID does not exist" Nov 01 20:16:54.709253 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.709214 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7"] Nov 01 20:16:54.711113 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:54.711090 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/e2e-conflict-a-b637b059-kserve-f5bf4947c-xjgq7"] Nov 01 20:16:55.157916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:16:55.157885 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" path="/var/lib/kubelet/pods/e3f5fec9-f6c2-46bc-b537-bf0ac92bc633/volumes" Nov 01 20:17:00.036569 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:17:00.036446 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:17:14.966740 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:14.966702 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2"] Nov 01 20:17:14.967285 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:14.967263 5444 cpu_manager.go:414] "containerMap: removing container" podUID="e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" containerName="main" Nov 01 20:17:14.967365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:14.967288 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" containerName="main" Nov 01 20:17:14.967437 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:14.967423 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f5fec9-f6c2-46bc-b537-bf0ac92bc633" containerName="main" Nov 01 20:17:14.970576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:14.970555 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.016013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.015976 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp"] Nov 01 20:17:15.019572 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.019551 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.068037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.067996 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2"] Nov 01 20:17:15.073717 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.073678 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp"] Nov 01 20:17:15.119450 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119405 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.119670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119462 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.119670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119568 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.119670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119620 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.119820 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119701 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.119820 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119770 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.119885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119827 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.119885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119853 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.119885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119874 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.119998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.119895 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.120049 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120024 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.120138 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120116 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.120180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120148 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm2b\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.120229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120184 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.120229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120216 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vr2t\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.120319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120307 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.120363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120329 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.120363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.120350 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221431 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221324 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221431 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221378 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.221431 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221419 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221468 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221490 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm2b\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221531 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221554 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vr2t\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221619 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221690 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221722 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.221786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221759 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.222143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221791 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.222143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221823 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.222143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221862 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.222143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221902 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.222143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.221928 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.222386 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222353 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.222463 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222446 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.222516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222502 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.222566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222531 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.222828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222806 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.222922 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222888 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.223143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.222501 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.223199 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.223141 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.223305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.223284 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.223427 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.223403 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.224665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.223957 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.224665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.224324 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.225510 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.225483 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.226233 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.226208 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.226335 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.226321 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.232291 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.232266 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.278327 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.278279 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.278523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.278280 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.278523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.278365 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm2b\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.278523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.278441 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vr2t\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.283990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.283955 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:15.333303 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.333262 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:15.711143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.711105 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2"] Nov 01 20:17:15.714984 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:17:15.714945 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62999052_8d11_4d48_b297_209dc4e6b4fe.slice/crio-fecf5d49bb42445538a9c7c5269770fc804a3b9720802ee3f141fe6d089116e9 WatchSource:0}: Error finding container fecf5d49bb42445538a9c7c5269770fc804a3b9720802ee3f141fe6d089116e9: Status 404 returned error can't find the container with id fecf5d49bb42445538a9c7c5269770fc804a3b9720802ee3f141fe6d089116e9 Nov 01 20:17:15.717774 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.717722 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:15.717926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.717822 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:15.717926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.717863 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:15.766137 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.766112 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp"] Nov 01 20:17:15.769062 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:17:15.769027 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d7659e_e0f4_4faa_937f_4fcc487c8dcb.slice/crio-186d24d108e0a3ae8931453e4c97e2543f76e2e82bfaf9e2296e5c28188e07c7 WatchSource:0}: Error finding container 186d24d108e0a3ae8931453e4c97e2543f76e2e82bfaf9e2296e5c28188e07c7: Status 404 returned error can't find the container with id 186d24d108e0a3ae8931453e4c97e2543f76e2e82bfaf9e2296e5c28188e07c7 Nov 01 20:17:15.771810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.771770 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:15.771936 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.771845 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:15.771936 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:15.771878 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:16.679659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:16.676315 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" event={"ID":"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb","Type":"ContainerStarted","Data":"dade670b1c60fcff8f647d8e6ca74b30ecde08c46d0000eab160e8d968fea6c4"} Nov 01 20:17:16.679659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:16.676387 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" event={"ID":"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb","Type":"ContainerStarted","Data":"186d24d108e0a3ae8931453e4c97e2543f76e2e82bfaf9e2296e5c28188e07c7"} Nov 01 20:17:16.679659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:16.679517 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" event={"ID":"62999052-8d11-4d48-b297-209dc4e6b4fe","Type":"ContainerStarted","Data":"cc45803bf68bbbfc23a63b3b7cc55f42844b788e3d6d463429b4f5627b3d7113"} Nov 01 20:17:16.679659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:16.679594 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" event={"ID":"62999052-8d11-4d48-b297-209dc4e6b4fe","Type":"ContainerStarted","Data":"fecf5d49bb42445538a9c7c5269770fc804a3b9720802ee3f141fe6d089116e9"} Nov 01 20:17:16.909056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:16.908998 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podStartSLOduration=2.908984374 podStartE2EDuration="2.908984374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:16.858254969 +0000 UTC m=+837.803092839" watchObservedRunningTime="2026-11-01 20:17:16.908984374 +0000 UTC m=+837.853822238" Nov 01 20:17:17.286349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:17.286314 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:17.286775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:17.286744 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 20:17:17.287176 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:17.287150 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 20:17:17.339355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:17.338997 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:17.339667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:17.339623 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 20:17:17.340307 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:17.340277 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 20:17:18.292568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:18.292104 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 20:17:18.292568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:18.292170 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 20:17:18.334199 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:18.334161 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 20:17:18.334391 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:18.334224 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 20:17:19.290143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:19.289053 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 20:17:19.290143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:19.289140 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 20:17:19.357694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:19.342420 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 20:17:19.357694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:19.342498 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 20:17:20.288672 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:20.288005 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 20:17:20.288672 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:20.288069 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 20:17:20.334587 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:20.334552 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 20:17:20.334773 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:20.334607 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 20:17:21.288099 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:21.288058 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 20:17:21.288775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:21.288740 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 20:17:21.333976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:21.333941 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 20:17:21.334139 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:21.333992 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 20:17:22.284970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:22.284926 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 20:17:22.285244 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:22.285010 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 20:17:22.334156 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:22.334111 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 20:17:22.334541 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:22.334182 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 20:17:23.289322 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.289290 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:23.289700 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.289677 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:23.290244 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.290227 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:17:23.338544 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.338515 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:23.339006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.338900 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:23.339675 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.339658 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:17:23.455775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:23.455714 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podStartSLOduration=9.455701382 podStartE2EDuration="9.455701382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:16.974785441 +0000 UTC m=+837.919623309" watchObservedRunningTime="2026-11-01 20:17:23.455701382 +0000 UTC m=+844.400539250" Nov 01 20:17:30.034212 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:17:30.034171 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:17:59.964423 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.964389 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj"] Nov 01 20:17:59.967551 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.967533 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982406 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982603 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982536 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982603 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982567 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmc8\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982603 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982594 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982618 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982812 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982783 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982853 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982818 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982853 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982838 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:17:59.982931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:17:59.982862 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.013997 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.013960 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx"] Nov 01 20:18:00.017631 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.017604 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.039784 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:18:00.039749 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:18:00.070860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.070824 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj"] Nov 01 20:18:00.084435 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084399 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.084435 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084436 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9k2s\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.084688 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084463 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.084688 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084524 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmc8\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.084688 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084607 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.084836 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084718 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.084836 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084746 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.084918 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084865 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.084979 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084961 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.085015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.084994 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085011 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.085217 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085193 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085240 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085291 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085270 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.085329 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085305 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085381 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085331 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.085381 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085351 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085545 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085522 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085610 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085562 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.085697 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085668 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085709 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085763 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.085931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.085912 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.087694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.087670 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.087898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.087881 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.141459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.141411 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.141674 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.141529 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmc8\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.171963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.171932 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx"] Nov 01 20:18:00.186333 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186293 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.186522 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186356 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.186522 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186410 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.186522 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186442 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.186522 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186494 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.186771 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186525 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.186951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186921 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.187036 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.186932 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.187036 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.187026 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.187143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.187105 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.187143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.187130 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9k2s\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.187246 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.187141 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.187594 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.187574 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.188080 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.188054 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.189019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.188994 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.189162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.189148 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.245257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.245181 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.245257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.245225 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9k2s\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s\") pod \"e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.280563 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.280528 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:00.333228 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.332772 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:00.706439 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.706397 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj"] Nov 01 20:18:00.747619 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:18:00.747587 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b2eff4_709a_47a6_92aa_632d962238b7.slice/crio-cad00dc129aa9b2983359f307e09ab5d06edcdb9b349d6c4e578fc05cb5bcafe WatchSource:0}: Error finding container cad00dc129aa9b2983359f307e09ab5d06edcdb9b349d6c4e578fc05cb5bcafe: Status 404 returned error can't find the container with id cad00dc129aa9b2983359f307e09ab5d06edcdb9b349d6c4e578fc05cb5bcafe Nov 01 20:18:00.749908 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.749872 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:18:00.749984 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.749946 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:18:00.750020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.749980 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:18:00.788263 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.785889 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx"] Nov 01 20:18:00.808184 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.808146 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:18:00.808319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.808241 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:18:00.808319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.808287 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:18:00.971020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.970921 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" event={"ID":"e0b2eff4-709a-47a6-92aa-632d962238b7","Type":"ContainerStarted","Data":"b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d"} Nov 01 20:18:00.971020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.970988 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" event={"ID":"e0b2eff4-709a-47a6-92aa-632d962238b7","Type":"ContainerStarted","Data":"cad00dc129aa9b2983359f307e09ab5d06edcdb9b349d6c4e578fc05cb5bcafe"} Nov 01 20:18:00.972711 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.972683 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" event={"ID":"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db","Type":"ContainerStarted","Data":"5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9"} Nov 01 20:18:00.972853 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:00.972728 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" event={"ID":"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db","Type":"ContainerStarted","Data":"1bcb9b37638a0145c435a373abf0ad98d3ff3f2321b4cd7ff5381cdef211d3f4"} Nov 01 20:18:01.087043 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.086991 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podStartSLOduration=2.086976158 podStartE2EDuration="2.086976158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:01.085127571 +0000 UTC m=+882.029965441" watchObservedRunningTime="2026-11-01 20:18:01.086976158 +0000 UTC m=+882.031814102" Nov 01 20:18:01.201115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.201058 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podStartSLOduration=2.201037481 podStartE2EDuration="2.201037481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:01.198956055 +0000 UTC m=+882.143793917" watchObservedRunningTime="2026-11-01 20:18:01.201037481 +0000 UTC m=+882.145875351" Nov 01 20:18:01.281236 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.281201 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:01.282497 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.282467 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:01.282632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.282516 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:01.333802 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.333759 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:01.335180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.335155 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:01.335301 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:01.335200 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:02.281663 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:02.281078 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:02.281663 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:02.281145 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:02.333453 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:02.333413 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:02.333610 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:02.333482 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:03.281180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:03.281140 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:03.281897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:03.281462 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:03.333902 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:03.333773 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:03.333902 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:03.333843 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:04.285659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:04.284041 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:04.285659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:04.284111 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:04.334371 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:04.334334 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:04.334554 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:04.334394 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:05.285662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:05.283072 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:05.285662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:05.283150 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:05.333426 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:05.333387 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:05.333586 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:05.333460 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:06.283096 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:06.283053 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:06.283446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:06.283417 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:06.333795 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:06.333759 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:06.334182 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:06.333828 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:07.282002 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:07.281966 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Nov 01 20:18:07.282293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:07.282255 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Nov 01 20:18:07.333817 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:07.333777 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 01 20:18:07.334235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:07.333858 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 01 20:18:08.287093 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:08.287053 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:08.287485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:08.287455 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:08.288481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:08.288459 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:18:08.593719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:08.592349 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:08.593719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:08.593709 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:08.599092 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:08.599060 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:18:24.019427 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.019386 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd"] Nov 01 20:18:24.024193 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.024170 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.112897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.112864 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5x4\" (UniqueName: \"kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.113081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.112920 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.113081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.113013 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.113081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.113072 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.113190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.113094 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.113190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.113177 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.127056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.126987 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7d5118\"/\"test-model-7d5118-kserve-self-signed-certs\"" Nov 01 20:18:24.127276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.127068 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7d5118\"/\"default-dockercfg-qx6rz\"" Nov 01 20:18:24.127276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.127140 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7d5118\"/\"kube-root-ca.crt\"" Nov 01 20:18:24.127276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.127264 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7d5118\"/\"openshift-service-ca.crt\"" Nov 01 20:18:24.128954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.128931 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd"] Nov 01 20:18:24.214998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.214876 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.214998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.214947 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.215349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.215026 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.215349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.215060 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.215349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.215110 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.215349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.215183 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5x4\" (UniqueName: \"kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.216116 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.216085 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.216401 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.216377 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.218764 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.218724 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.222526 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.222433 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.223030 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.223003 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.272850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.272768 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5x4\" (UniqueName: \"kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4\") pod \"test-model-7d5118-kserve-6779d5dfd7-qxzrd\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.335717 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.335670 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:24.727729 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:24.727703 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd"] Nov 01 20:18:24.730433 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:18:24.730402 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c5ab51a_3139_42dd_af1d_2230e046c9f0.slice/crio-8a9bcd6215b18f13f6c4d4fac56fe044d7ffb4dc9d72f751846bc9fcee8ec72f WatchSource:0}: Error finding container 8a9bcd6215b18f13f6c4d4fac56fe044d7ffb4dc9d72f751846bc9fcee8ec72f: Status 404 returned error can't find the container with id 8a9bcd6215b18f13f6c4d4fac56fe044d7ffb4dc9d72f751846bc9fcee8ec72f Nov 01 20:18:25.106550 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:25.106500 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" event={"ID":"0c5ab51a-3139-42dd-af1d-2230e046c9f0","Type":"ContainerStarted","Data":"d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee"} Nov 01 20:18:25.106550 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:25.106555 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" event={"ID":"0c5ab51a-3139-42dd-af1d-2230e046c9f0","Type":"ContainerStarted","Data":"8a9bcd6215b18f13f6c4d4fac56fe044d7ffb4dc9d72f751846bc9fcee8ec72f"} Nov 01 20:18:25.220300 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:25.220245 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" podStartSLOduration=2.220228019 podStartE2EDuration="2.220228019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:25.218203247 +0000 UTC m=+906.163041116" watchObservedRunningTime="2026-11-01 20:18:25.220228019 +0000 UTC m=+906.165065887" Nov 01 20:18:25.351773 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:25.351735 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:30.038220 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:18:30.038177 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:18:34.336060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:34.336013 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:34.348950 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:34.348925 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:35.183833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:35.183803 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:18:57.541417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:57.541383 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx"] Nov 01 20:18:57.541910 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:18:57.541604 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" containerID="cri-o://8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872" gracePeriod=30 Nov 01 20:19:00.035485 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:19:00.035446 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:19:02.791264 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.791239 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858279 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858388 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858475 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858505 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858539 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858568 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858620 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858745 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.859753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.858810 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht\") pod \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\" (UID: \"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe\") " Nov 01 20:19:02.860410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.860323 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:19:02.860745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.860536 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:19:02.860844 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.860777 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:19:02.861367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.861003 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:19:02.861367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.861296 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:02.861977 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.861931 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "kube-api-access-v44ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:02.887734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.887677 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:02.888331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.888292 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:19:02.889482 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.889433 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo" pod "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" (UID: "6c36d04f-fb39-4ac1-95f8-88a2767c5cfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:19:02.960649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960608 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960646 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v44ht\" (UniqueName: \"kubernetes.io/projected/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-kube-api-access-v44ht\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960658 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960668 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960676 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960684 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960692 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960700 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.960904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:02.960708 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:19:03.400071 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.400035 5444 generic.go:356] "Generic (PLEG): container finished" podID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerID="8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872" exitCode=0 Nov 01 20:19:03.400257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.400097 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" Nov 01 20:19:03.400257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.400114 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" event={"ID":"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe","Type":"ContainerDied","Data":"8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872"} Nov 01 20:19:03.400257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.400191 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx" event={"ID":"6c36d04f-fb39-4ac1-95f8-88a2767c5cfe","Type":"ContainerDied","Data":"06dc357211d9092d37cac2b180e5ca917cdd3b369096ec8daa00dd592ff758bb"} Nov 01 20:19:03.400257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.400222 5444 scope.go:122] "RemoveContainer" containerID="8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872" Nov 01 20:19:03.408884 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.408862 5444 scope.go:122] "RemoveContainer" containerID="8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872" Nov 01 20:19:03.409147 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:19:03.409127 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872\": container with ID starting with 8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872 not found: ID does not exist" containerID="8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872" Nov 01 20:19:03.409201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.409156 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872"} err="failed to get container status \"8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872\": rpc error: code = NotFound desc = could not find container \"8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872\": container with ID starting with 8cdc1691a5781f9ad4de83d043954c4a93257570d0074500fd552012e9aa2872 not found: ID does not exist" Nov 01 20:19:03.562437 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.562407 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx"] Nov 01 20:19:03.566289 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:03.566266 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qrwgx"] Nov 01 20:19:05.158319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:05.158291 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" path="/var/lib/kubelet/pods/6c36d04f-fb39-4ac1-95f8-88a2767c5cfe/volumes" Nov 01 20:19:14.254405 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.254370 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4"] Nov 01 20:19:14.254818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.254791 5444 cpu_manager.go:414] "containerMap: removing container" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" Nov 01 20:19:14.254818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.254805 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" Nov 01 20:19:14.254902 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.254889 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c36d04f-fb39-4ac1-95f8-88a2767c5cfe" containerName="istio-proxy" Nov 01 20:19:14.259364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.259343 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.311288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.311266 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-df41b8-openshift-default-dockercfg-75q9g\"" Nov 01 20:19:14.349609 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349580 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmrr\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.349737 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349620 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.349737 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349701 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.349836 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349799 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.349836 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349830 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.349926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349849 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.349926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349908 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.350006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.349938 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.350175 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.350151 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.363323 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.363300 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4"] Nov 01 20:19:14.450734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.450704 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.450846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.450743 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.450846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.450771 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.450846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.450810 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.450875 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.450979 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmrr\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451123 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451013 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451123 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451039 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451123 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451101 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451160 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451168 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451212 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451446 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.451807 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.451786 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.453290 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.453268 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.453499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.453480 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.506648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.506619 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.506745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.506725 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmrr\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.571714 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.571696 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:14.996755 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:14.996672 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4"] Nov 01 20:19:14.999436 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:19:14.999411 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7edbe4e_7819_4702_a5e4_77ca37e0004a.slice/crio-392622527f0d66c707a60a36aeb263c6b2b0d1d5d3f4123f0f13209663c3d6aa WatchSource:0}: Error finding container 392622527f0d66c707a60a36aeb263c6b2b0d1d5d3f4123f0f13209663c3d6aa: Status 404 returned error can't find the container with id 392622527f0d66c707a60a36aeb263c6b2b0d1d5d3f4123f0f13209663c3d6aa Nov 01 20:19:15.001568 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:15.001531 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:19:15.001670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:15.001598 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:19:15.001670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:15.001627 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:19:15.575744 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:15.575709 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" event={"ID":"d7edbe4e-7819-4702-a5e4-77ca37e0004a","Type":"ContainerStarted","Data":"92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02"} Nov 01 20:19:15.576142 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:15.575760 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" event={"ID":"d7edbe4e-7819-4702-a5e4-77ca37e0004a","Type":"ContainerStarted","Data":"392622527f0d66c707a60a36aeb263c6b2b0d1d5d3f4123f0f13209663c3d6aa"} Nov 01 20:19:15.690791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:15.690732 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podStartSLOduration=1.69071266 podStartE2EDuration="1.69071266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:15.68971711 +0000 UTC m=+956.634554989" watchObservedRunningTime="2026-11-01 20:19:15.69071266 +0000 UTC m=+956.635550531" Nov 01 20:19:16.576663 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:16.575922 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:16.585081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:16.580754 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 20:19:16.585081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:16.580919 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 20:19:17.576135 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:17.576097 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 20:19:17.576461 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:17.576430 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 20:19:18.574091 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:18.574053 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 20:19:18.574702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:18.574667 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 20:19:19.572155 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:19.572122 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 20:19:19.572321 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:19.572201 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 20:19:20.572043 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:20.572007 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 01 20:19:20.572453 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:20.572058 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 01 20:19:21.876316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:21.876286 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:21.878977 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:21.878943 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:21.895191 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:19:21.895164 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:19:30.037854 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:19:30.037815 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:20:00.034695 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:20:00.034592 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:20:30.035577 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:20:30.035539 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:20:53.556751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.556713 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq"] Nov 01 20:20:53.560374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.560357 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.612378 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.612350 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7d5118-openshift-default-dockercfg-2rtxd\"" Nov 01 20:20:53.641571 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641545 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gsr\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641607 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641706 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641731 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641755 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641814 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641878 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641843 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641921 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.641975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.641954 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.664490 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.664465 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq"] Nov 01 20:20:53.742534 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742505 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.742685 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742541 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.742748 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742683 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gsr\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.742808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742788 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.742860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742835 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.742918 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742867 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.742918 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742904 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742942 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.742971 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743318 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.743292 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743318 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.743309 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743445 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.743321 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743445 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.743387 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.743445 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.743416 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.745035 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.745013 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.745136 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.745111 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.798365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.798342 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.798557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.798539 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gsr\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr\") pod \"e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:53.872726 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:53.872673 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:54.307821 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:54.307794 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq"] Nov 01 20:20:54.311075 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:20:54.311049 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640182a1_8f46_425e_a287_3709bcdc82e5.slice/crio-9fd472507fe4fba9bed465cf945606500bd7f4e7111dd0dc7d91d4a80e410598 WatchSource:0}: Error finding container 9fd472507fe4fba9bed465cf945606500bd7f4e7111dd0dc7d91d4a80e410598: Status 404 returned error can't find the container with id 9fd472507fe4fba9bed465cf945606500bd7f4e7111dd0dc7d91d4a80e410598 Nov 01 20:20:54.313165 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:54.313130 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:20:54.313242 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:54.313196 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:20:54.313281 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:54.313239 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:20:55.291742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:55.291705 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" event={"ID":"640182a1-8f46-425e-a287-3709bcdc82e5","Type":"ContainerStarted","Data":"489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae"} Nov 01 20:20:55.292182 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:55.291758 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" event={"ID":"640182a1-8f46-425e-a287-3709bcdc82e5","Type":"ContainerStarted","Data":"9fd472507fe4fba9bed465cf945606500bd7f4e7111dd0dc7d91d4a80e410598"} Nov 01 20:20:55.511838 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:55.511779 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podStartSLOduration=2.511765265 podStartE2EDuration="2.511765265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:55.46113892 +0000 UTC m=+1056.405976790" watchObservedRunningTime="2026-11-01 20:20:55.511765265 +0000 UTC m=+1056.456603133" Nov 01 20:20:55.873164 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:55.873072 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:20:55.874562 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:55.874534 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 20:20:55.874723 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:55.874584 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 20:20:56.874037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:56.874000 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 20:20:56.875477 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:56.875444 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 20:20:57.877816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:57.876080 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 20:20:57.877816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:57.876156 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 20:20:58.876073 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:58.876035 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 20:20:58.876403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:58.876360 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 20:20:59.877659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:59.876092 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 20:20:59.877659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:20:59.876155 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 20:21:00.112803 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:21:00.112667 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:21:00.873033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:21:00.872994 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 01 20:21:00.873346 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:21:00.873059 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 01 20:21:01.904284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:21:01.902809 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:21:01.904284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:21:01.904127 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:21:01.905953 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:21:01.905902 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:21:30.034553 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:21:30.034515 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:22:00.034573 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:22:00.034536 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:22:07.059182 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:07.059140 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd"] Nov 01 20:22:22.078392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.078328 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" podUID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" containerName="main" containerID="cri-o://d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee" gracePeriod=45 Nov 01 20:22:22.173955 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.173924 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" podUID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": dial tcp 10.133.0.46:8000: connect: connection refused" Nov 01 20:22:22.332148 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.332097 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:22:22.503204 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503176 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home\") pod \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " Nov 01 20:22:22.503334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503214 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs\") pod \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " Nov 01 20:22:22.503334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503268 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm\") pod \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " Nov 01 20:22:22.503334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503290 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache\") pod \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " Nov 01 20:22:22.503479 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503352 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4\" (UniqueName: \"kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4\") pod \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " Nov 01 20:22:22.503479 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503378 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir\") pod \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\" (UID: \"0c5ab51a-3139-42dd-af1d-2230e046c9f0\") " Nov 01 20:22:22.503479 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503403 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home" pod "0c5ab51a-3139-42dd-af1d-2230e046c9f0" (UID: "0c5ab51a-3139-42dd-af1d-2230e046c9f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:22.503667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503625 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:22.503731 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503676 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache" pod "0c5ab51a-3139-42dd-af1d-2230e046c9f0" (UID: "0c5ab51a-3139-42dd-af1d-2230e046c9f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:22.503731 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.503714 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir" pod "0c5ab51a-3139-42dd-af1d-2230e046c9f0" (UID: "0c5ab51a-3139-42dd-af1d-2230e046c9f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:22.505527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.505502 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm" pod "0c5ab51a-3139-42dd-af1d-2230e046c9f0" (UID: "0c5ab51a-3139-42dd-af1d-2230e046c9f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:22.505651 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.505526 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4" pod "0c5ab51a-3139-42dd-af1d-2230e046c9f0" (UID: "0c5ab51a-3139-42dd-af1d-2230e046c9f0"). InnerVolumeSpecName "kube-api-access-cm5x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:22.505651 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.505534 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs" pod "0c5ab51a-3139-42dd-af1d-2230e046c9f0" (UID: "0c5ab51a-3139-42dd-af1d-2230e046c9f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:22.604456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.604395 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5x4\" (UniqueName: \"kubernetes.io/projected/0c5ab51a-3139-42dd-af1d-2230e046c9f0-kube-api-access-cm5x4\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:22.604456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.604420 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:22.604456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.604434 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5ab51a-3139-42dd-af1d-2230e046c9f0-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:22.604456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.604446 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:22.604675 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.604459 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5ab51a-3139-42dd-af1d-2230e046c9f0-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:22.873874 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.873806 5444 generic.go:356] "Generic (PLEG): container finished" podID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" containerID="d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee" exitCode=0 Nov 01 20:22:22.873874 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.873845 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" Nov 01 20:22:22.873874 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.873862 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" event={"ID":"0c5ab51a-3139-42dd-af1d-2230e046c9f0","Type":"ContainerDied","Data":"d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee"} Nov 01 20:22:22.874109 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.873892 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd" event={"ID":"0c5ab51a-3139-42dd-af1d-2230e046c9f0","Type":"ContainerDied","Data":"8a9bcd6215b18f13f6c4d4fac56fe044d7ffb4dc9d72f751846bc9fcee8ec72f"} Nov 01 20:22:22.874109 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.873908 5444 scope.go:122] "RemoveContainer" containerID="d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee" Nov 01 20:22:22.882515 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.882502 5444 scope.go:122] "RemoveContainer" containerID="d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee" Nov 01 20:22:22.882797 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:22:22.882776 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee\": container with ID starting with d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee not found: ID does not exist" containerID="d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee" Nov 01 20:22:22.882862 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:22.882806 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee"} err="failed to get container status \"d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee\": rpc error: code = NotFound desc = could not find container \"d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee\": container with ID starting with d9d410fe43fbd74bdf09e1140c15d5f6c1e68ff91888043ccb4b0d072893d3ee not found: ID does not exist" Nov 01 20:22:23.035328 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:23.035305 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd"] Nov 01 20:22:23.084959 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:23.084933 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7d5118/test-model-7d5118-kserve-6779d5dfd7-qxzrd"] Nov 01 20:22:23.158667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:23.158593 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" path="/var/lib/kubelet/pods/0c5ab51a-3139-42dd-af1d-2230e046c9f0/volumes" Nov 01 20:22:30.040151 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:22:30.040111 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:22:33.237233 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.237203 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq"] Nov 01 20:22:33.237606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.237593 5444 cpu_manager.go:414] "containerMap: removing container" podUID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" containerName="main" Nov 01 20:22:33.237676 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.237606 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" containerName="main" Nov 01 20:22:33.237716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.237706 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5ab51a-3139-42dd-af1d-2230e046c9f0" containerName="main" Nov 01 20:22:33.240803 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.240785 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.320758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.320731 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.320911 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.320770 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.320911 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.320794 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.320998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.320943 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.320998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.320976 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.321071 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.321057 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs24d\" (UniqueName: \"kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.342817 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.342797 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"kube-root-ca.crt\"" Nov 01 20:22:33.342817 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.342802 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"rate-test-model-2cc24d-kserve-self-signed-certs\"" Nov 01 20:22:33.344440 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.344422 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq"] Nov 01 20:22:33.418037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.418002 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"openshift-service-ca.crt\"" Nov 01 20:22:33.442110 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.420945 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"default-dockercfg-2pqzm\"" Nov 01 20:22:33.442347 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.425511 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs24d\" (UniqueName: \"kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.442567 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.442546 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.443164 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.443144 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.443356 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.443336 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.444181 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.444159 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.455162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.449714 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.455162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.443491 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.455162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.443069 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.455162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.452837 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.455162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.453361 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.456038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.455996 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.487430 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.487364 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs24d\" (UniqueName: \"kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d\") pod \"rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.552147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.552118 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:33.932040 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:33.931936 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq"] Nov 01 20:22:33.935337 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:22:33.935303 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb7d1c0_9a82_47f4_89a2_e03ebd9a2eda.slice/crio-43d99f86e6f3278559bee00b167367823346d195a40bc8c2c3f554e2baf29783 WatchSource:0}: Error finding container 43d99f86e6f3278559bee00b167367823346d195a40bc8c2c3f554e2baf29783: Status 404 returned error can't find the container with id 43d99f86e6f3278559bee00b167367823346d195a40bc8c2c3f554e2baf29783 Nov 01 20:22:34.942293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:34.942262 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" event={"ID":"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda","Type":"ContainerStarted","Data":"170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24"} Nov 01 20:22:34.942293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:34.942300 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" event={"ID":"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda","Type":"ContainerStarted","Data":"43d99f86e6f3278559bee00b167367823346d195a40bc8c2c3f554e2baf29783"} Nov 01 20:22:35.056687 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:35.056618 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" podStartSLOduration=2.056604385 podStartE2EDuration="2.056604385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:35.05650379 +0000 UTC m=+1156.001341660" watchObservedRunningTime="2026-11-01 20:22:35.056604385 +0000 UTC m=+1156.001442253" Nov 01 20:22:35.569665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:35.560088 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:43.556662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:43.555619 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:43.667660 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:43.666778 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:44.302273 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:44.302015 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:22:50.622574 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:50.622496 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz"] Nov 01 20:22:50.623017 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:50.622723 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" podUID="700a1795-58fc-4939-aba9-2f7c070d136b" containerName="maas-api" containerID="cri-o://78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733" gracePeriod=30 Nov 01 20:22:50.962716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:50.962689 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:22:51.068860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.068828 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs\") pod \"700a1795-58fc-4939-aba9-2f7c070d136b\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " Nov 01 20:22:51.069060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.068908 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\") pod \"700a1795-58fc-4939-aba9-2f7c070d136b\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " Nov 01 20:22:51.069060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.069024 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca\") pod \"700a1795-58fc-4939-aba9-2f7c070d136b\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " Nov 01 20:22:51.069178 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.069077 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb\" (UniqueName: \"kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb\") pod \"700a1795-58fc-4939-aba9-2f7c070d136b\" (UID: \"700a1795-58fc-4939-aba9-2f7c070d136b\") " Nov 01 20:22:51.069359 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.069329 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca" pod "700a1795-58fc-4939-aba9-2f7c070d136b" (UID: "700a1795-58fc-4939-aba9-2f7c070d136b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:51.069519 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.069501 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700a1795-58fc-4939-aba9-2f7c070d136b-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:51.070284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.070261 5444 generic.go:356] "Generic (PLEG): container finished" podID="700a1795-58fc-4939-aba9-2f7c070d136b" containerID="78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733" exitCode=0 Nov 01 20:22:51.070397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.070306 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" Nov 01 20:22:51.070397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.070320 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" event={"ID":"700a1795-58fc-4939-aba9-2f7c070d136b","Type":"ContainerDied","Data":"78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733"} Nov 01 20:22:51.070397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.070365 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz" event={"ID":"700a1795-58fc-4939-aba9-2f7c070d136b","Type":"ContainerDied","Data":"06f51cf6ee8f2fcc2868e5a1417d2bc2490c3206b1b906c0165159b51f6d8bf1"} Nov 01 20:22:51.070397 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.070391 5444 scope.go:122] "RemoveContainer" containerID="78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733" Nov 01 20:22:51.071414 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.071394 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs" pod "700a1795-58fc-4939-aba9-2f7c070d136b" (UID: "700a1795-58fc-4939-aba9-2f7c070d136b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:51.072694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.072625 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls" pod "700a1795-58fc-4939-aba9-2f7c070d136b" (UID: "700a1795-58fc-4939-aba9-2f7c070d136b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:51.076997 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.076971 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb" pod "700a1795-58fc-4939-aba9-2f7c070d136b" (UID: "700a1795-58fc-4939-aba9-2f7c070d136b"). InnerVolumeSpecName "kube-api-access-mc9cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:51.092916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.092895 5444 scope.go:122] "RemoveContainer" containerID="78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733" Nov 01 20:22:51.093235 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:22:51.093213 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733\": container with ID starting with 78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733 not found: ID does not exist" containerID="78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733" Nov 01 20:22:51.093319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.093242 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733"} err="failed to get container status \"78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733\": rpc error: code = NotFound desc = could not find container \"78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733\": container with ID starting with 78447cf939001f8ed4c19371f43c81b72d78c8b6ba25384779bf4d8a3d862733 not found: ID does not exist" Nov 01 20:22:51.170672 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.170626 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:51.170956 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.170674 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700a1795-58fc-4939-aba9-2f7c070d136b-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:51.170956 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.170690 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9cb\" (UniqueName: \"kubernetes.io/projected/700a1795-58fc-4939-aba9-2f7c070d136b-kube-api-access-mc9cb\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:22:51.483262 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.483232 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz"] Nov 01 20:22:51.569664 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:51.562735 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ad7edb-7b749bfbdf-vdngz"] Nov 01 20:22:53.157340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:22:53.157310 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700a1795-58fc-4939-aba9-2f7c070d136b" path="/var/lib/kubelet/pods/700a1795-58fc-4939-aba9-2f7c070d136b/volumes" Nov 01 20:23:00.045660 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:23:00.045010 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:23:17.466788 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:17.465735 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:17.578775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:17.578715 5444 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" podUID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:23:17.626824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:17.626746 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 20:23:30.035016 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:23:30.034980 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:23:44.731332 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:44.731302 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk"] Nov 01 20:23:44.731719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:44.731525 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" containerID="cri-o://c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568" gracePeriod=30 Nov 01 20:23:46.518402 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:46.518372 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq"] Nov 01 20:23:46.518791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:46.518568 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" containerID="cri-o://489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae" gracePeriod=30 Nov 01 20:23:46.906371 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:46.906288 5444 patch_prober.go:29] interesting pod/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:23:46.906371 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:46.906349 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:23:49.245295 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:49.245266 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kc6jk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:23:49.245775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:49.245325 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:23:49.983350 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:49.983328 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:23:50.056872 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.056847 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.056990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.056904 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.056990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.056935 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.056990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.056975 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.057141 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057118 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.057208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057172 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.057266 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057206 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.057319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057295 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.057376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057323 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:50.057376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057362 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert\") pod \"73cc12c9-6da4-43eb-9f66-01842db9605b\" (UID: \"73cc12c9-6da4-43eb-9f66-01842db9605b\") " Nov 01 20:23:50.057491 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057474 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:50.057543 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057488 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:50.057797 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057751 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:50.057892 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057871 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.057892 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057886 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:50.058006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057892 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.058006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057916 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.058006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.057931 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.059042 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.059014 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:50.059149 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.059111 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "kube-api-access-fbz96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:50.059415 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.059395 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:23:50.059484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.059448 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token" pod "73cc12c9-6da4-43eb-9f66-01842db9605b" (UID: "73cc12c9-6da4-43eb-9f66-01842db9605b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:50.159403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.159377 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cc12c9-6da4-43eb-9f66-01842db9605b-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.159403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.159398 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.159403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.159408 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.159628 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.159417 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbz96\" (UniqueName: \"kubernetes.io/projected/73cc12c9-6da4-43eb-9f66-01842db9605b-kube-api-access-fbz96\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.159628 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.159426 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cc12c9-6da4-43eb-9f66-01842db9605b-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:50.441698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.441601 5444 generic.go:356] "Generic (PLEG): container finished" podID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerID="c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568" exitCode=0 Nov 01 20:23:50.441698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.441672 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" Nov 01 20:23:50.441698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.441681 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" event={"ID":"73cc12c9-6da4-43eb-9f66-01842db9605b","Type":"ContainerDied","Data":"c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568"} Nov 01 20:23:50.442176 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.441735 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk" event={"ID":"73cc12c9-6da4-43eb-9f66-01842db9605b","Type":"ContainerDied","Data":"5fa18e7153bb1b82b948521d668d0902d0cfd0264b39402fbd7671d3189a627d"} Nov 01 20:23:50.442176 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.441757 5444 scope.go:122] "RemoveContainer" containerID="c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568" Nov 01 20:23:50.451831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.451809 5444 scope.go:122] "RemoveContainer" containerID="c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568" Nov 01 20:23:50.452140 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:23:50.452121 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568\": container with ID starting with c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568 not found: ID does not exist" containerID="c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568" Nov 01 20:23:50.452216 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.452151 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568"} err="failed to get container status \"c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568\": rpc error: code = NotFound desc = could not find container \"c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568\": container with ID starting with c030f8662abcd00bdba3ae80a5a324b288a72b893c18898916248857993f9568 not found: ID does not exist" Nov 01 20:23:50.603789 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.603764 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk"] Nov 01 20:23:50.653369 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:50.653343 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kc6jk"] Nov 01 20:23:51.157969 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.157941 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" path="/var/lib/kubelet/pods/73cc12c9-6da4-43eb-9f66-01842db9605b/volumes" Nov 01 20:23:51.771376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.771352 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:23:51.873087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873036 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873066 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873085 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873109 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873128 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873231 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873282 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873318 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873349 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data\") pod \"640182a1-8f46-425e-a287-3709bcdc82e5\" (UID: \"640182a1-8f46-425e-a287-3709bcdc82e5\") " Nov 01 20:23:51.873540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873387 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.873540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873466 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.873540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873480 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:51.873845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873702 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.873845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873756 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.873845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873760 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.873845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873777 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/640182a1-8f46-425e-a287-3709bcdc82e5-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.873845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.873792 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.875273 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.875246 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "kube-api-access-x2gsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:51.875364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.875333 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:23:51.875632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.875600 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:23:51.875632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.875608 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token" pod "640182a1-8f46-425e-a287-3709bcdc82e5" (UID: "640182a1-8f46-425e-a287-3709bcdc82e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:51.974391 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.974366 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.974391 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.974390 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.974523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.974399 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.974523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.974408 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gsr\" (UniqueName: \"kubernetes.io/projected/640182a1-8f46-425e-a287-3709bcdc82e5-kube-api-access-x2gsr\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.974523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.974421 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/640182a1-8f46-425e-a287-3709bcdc82e5-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:51.974523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:51.974433 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/640182a1-8f46-425e-a287-3709bcdc82e5-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:23:52.453616 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.453583 5444 generic.go:356] "Generic (PLEG): container finished" podID="640182a1-8f46-425e-a287-3709bcdc82e5" containerID="489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae" exitCode=0 Nov 01 20:23:52.453814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.453659 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" Nov 01 20:23:52.453814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.453664 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" event={"ID":"640182a1-8f46-425e-a287-3709bcdc82e5","Type":"ContainerDied","Data":"489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae"} Nov 01 20:23:52.453814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.453718 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq" event={"ID":"640182a1-8f46-425e-a287-3709bcdc82e5","Type":"ContainerDied","Data":"9fd472507fe4fba9bed465cf945606500bd7f4e7111dd0dc7d91d4a80e410598"} Nov 01 20:23:52.453814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.453741 5444 scope.go:122] "RemoveContainer" containerID="489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae" Nov 01 20:23:52.462964 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.462947 5444 scope.go:122] "RemoveContainer" containerID="489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae" Nov 01 20:23:52.463225 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:23:52.463204 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae\": container with ID starting with 489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae not found: ID does not exist" containerID="489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae" Nov 01 20:23:52.463274 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.463236 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae"} err="failed to get container status \"489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae\": rpc error: code = NotFound desc = could not find container \"489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae\": container with ID starting with 489dfbe30d7f4fd96d854b5da594c1660e12aa2d9ad0e179257834634abe34ae not found: ID does not exist" Nov 01 20:23:52.664445 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.664421 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq"] Nov 01 20:23:52.713826 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:52.713779 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7d5118-openshift-default-5dbf75d645-whwkq"] Nov 01 20:23:53.157742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:23:53.157713 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" path="/var/lib/kubelet/pods/640182a1-8f46-425e-a287-3709bcdc82e5/volumes" Nov 01 20:24:00.035162 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:24:00.035125 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:24:30.034938 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:24:30.034905 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:24:39.336663 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.335831 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:39.336663 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.335937 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:39.463757 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.463713 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:39.645931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.645842 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:39.646210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.646178 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:39.676950 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.676911 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:39.704753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.704719 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:24:39.705006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:39.704978 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:24:41.827166 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827133 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z"] Nov 01 20:24:41.827559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827501 5444 cpu_manager.go:414] "containerMap: removing container" podUID="700a1795-58fc-4939-aba9-2f7c070d136b" containerName="maas-api" Nov 01 20:24:41.827559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827511 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a1795-58fc-4939-aba9-2f7c070d136b" containerName="maas-api" Nov 01 20:24:41.827559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827522 5444 cpu_manager.go:414] "containerMap: removing container" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" Nov 01 20:24:41.827559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827527 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" Nov 01 20:24:41.827559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827546 5444 cpu_manager.go:414] "containerMap: removing container" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" Nov 01 20:24:41.827559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827550 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" Nov 01 20:24:41.827832 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827614 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a1795-58fc-4939-aba9-2f7c070d136b" containerName="maas-api" Nov 01 20:24:41.827832 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827624 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="640182a1-8f46-425e-a287-3709bcdc82e5" containerName="istio-proxy" Nov 01 20:24:41.827832 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.827632 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cc12c9-6da4-43eb-9f66-01842db9605b" containerName="istio-proxy" Nov 01 20:24:41.830703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.830688 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.882662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.882614 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2cc24d-openshift-default-dockercfg-b4lxl\"" Nov 01 20:24:41.935219 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.935190 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z"] Nov 01 20:24:41.959352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959317 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959365 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959458 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959484 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959706 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959501 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959706 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959625 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959702 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959739 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxpd\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:41.959818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:41.959781 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.060795 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.060763 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.060996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.060810 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.060996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.060848 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.060996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.060873 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.060996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.060898 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.060996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.060984 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061221 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061039 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061221 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061071 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxpd\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061221 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061097 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061563 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061754 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061650 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061754 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061741 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061853 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061812 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.061954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.061849 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.063617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.063590 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.063780 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.063762 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.117229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.117161 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.117347 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.117308 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxpd\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.143153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.143133 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:42.564424 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.564395 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z"] Nov 01 20:24:42.567094 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:24:42.567063 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4c55be_d563_4731_bb10_9cabb14c0c73.slice/crio-a85790fd4eddd3c6dacfc19868d758cfae8da40b4ef1fafc7e49d8257e2422ae WatchSource:0}: Error finding container a85790fd4eddd3c6dacfc19868d758cfae8da40b4ef1fafc7e49d8257e2422ae: Status 404 returned error can't find the container with id a85790fd4eddd3c6dacfc19868d758cfae8da40b4ef1fafc7e49d8257e2422ae Nov 01 20:24:42.569373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.569340 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:24:42.569453 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.569406 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:24:42.569453 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.569436 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:24:42.847315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.847231 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" event={"ID":"2d4c55be-d563-4731-bb10-9cabb14c0c73","Type":"ContainerStarted","Data":"9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325"} Nov 01 20:24:42.847315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.847279 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" event={"ID":"2d4c55be-d563-4731-bb10-9cabb14c0c73","Type":"ContainerStarted","Data":"a85790fd4eddd3c6dacfc19868d758cfae8da40b4ef1fafc7e49d8257e2422ae"} Nov 01 20:24:42.962566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:42.962508 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podStartSLOduration=1.962491418 podStartE2EDuration="1.962491418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:42.961103739 +0000 UTC m=+1283.905941618" watchObservedRunningTime="2026-11-01 20:24:42.962491418 +0000 UTC m=+1283.907329287" Nov 01 20:24:43.143406 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:43.143319 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:43.144412 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:43.144390 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 01 20:24:43.144514 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:43.144434 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 01 20:24:44.144347 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:44.144311 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 01 20:24:44.144859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:44.144366 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 01 20:24:45.143904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:45.143874 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 01 20:24:45.144087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:45.143921 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 01 20:24:46.144072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:46.144039 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 01 20:24:46.144564 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:46.144086 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 01 20:24:47.144240 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:47.144200 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 01 20:24:47.144618 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:47.144258 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 01 20:24:48.143986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:48.143955 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 01 20:24:48.144150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:48.144005 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 01 20:24:49.148028 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:49.147998 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:49.148444 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:49.148295 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:24:49.149080 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:24:49.149059 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:25:00.038255 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:25:00.038213 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:25:30.052985 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:25:30.052951 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:25:44.339844 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.339806 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc"] Nov 01 20:25:44.343368 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.343346 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.395049 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.395027 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71335882-serving-cert\"" Nov 01 20:25:44.442506 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.442479 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.442685 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.442531 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.442685 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.442546 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrsw\" (UniqueName: \"kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.442816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.442777 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.446194 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.446173 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc"] Nov 01 20:25:44.543257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.543223 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.543392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.543275 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.543392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.543293 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrsw\" (UniqueName: \"kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.543392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.543356 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.543392 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:25:44.543370 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-71335882-serving-cert: secret "maas-api-e2e-mt-71335882-serving-cert" not found Nov 01 20:25:44.543571 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:25:44.543425 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls podName:ab46a205-b355-4ad8-ad32-e61386b2014a nodeName:}" failed. No retries permitted until 2026-11-01 20:25:45.043410826 +0000 UTC m=+1345.988248671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls") pod "maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" (UID: "ab46a205-b355-4ad8-ad32-e61386b2014a") : secret "maas-api-e2e-mt-71335882-serving-cert" not found Nov 01 20:25:44.543896 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.543874 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.545746 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.545727 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:44.608703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:44.608120 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrsw\" (UniqueName: \"kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:45.052792 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:45.052759 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:45.055080 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:45.055060 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:45.256517 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:45.256491 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:45.628432 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:45.628362 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc"] Nov 01 20:25:45.632789 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:25:45.632748 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab46a205_b355_4ad8_ad32_e61386b2014a.slice/crio-d87779f42e6e6af695560b5b6b6f41abc4078b0537f71dac0574f0c6f3a86d08 WatchSource:0}: Error finding container d87779f42e6e6af695560b5b6b6f41abc4078b0537f71dac0574f0c6f3a86d08: Status 404 returned error can't find the container with id d87779f42e6e6af695560b5b6b6f41abc4078b0537f71dac0574f0c6f3a86d08 Nov 01 20:25:45.633465 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:45.633446 5444 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:25:46.425959 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:46.425917 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" event={"ID":"ab46a205-b355-4ad8-ad32-e61386b2014a","Type":"ContainerStarted","Data":"d87779f42e6e6af695560b5b6b6f41abc4078b0537f71dac0574f0c6f3a86d08"} Nov 01 20:25:47.432285 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:47.432253 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" event={"ID":"ab46a205-b355-4ad8-ad32-e61386b2014a","Type":"ContainerStarted","Data":"b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a"} Nov 01 20:25:47.432692 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:47.432371 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:25:47.546509 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:47.546452 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" podStartSLOduration=2.66154457 podStartE2EDuration="3.546434203s" totalImagesPullingTime="884.889633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:47.544268822 +0000 UTC m=+1348.489106691" watchObservedRunningTime="2026-11-01 20:25:47.546434203 +0000 UTC m=+1348.491272072" Nov 01 20:25:53.441435 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:25:53.441359 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:26:00.041828 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:26:00.041788 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:26:30.037111 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:26:30.037069 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:26:50.355256 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.355222 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn"] Nov 01 20:26:50.359519 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.359498 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.413260 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.413234 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn"] Nov 01 20:26:50.460034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.460008 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-46c731c3-serving-cert\"" Nov 01 20:26:50.486219 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.486194 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5ch\" (UniqueName: \"kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.486319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.486247 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.486319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.486287 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.486452 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.486410 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.587462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.587437 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5ch\" (UniqueName: \"kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.587605 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.587486 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.587605 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.587505 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.587771 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:26:50.587606 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-serving-cert: secret "maas-api-e2e-migrate-46c731c3-serving-cert" not found Nov 01 20:26:50.587771 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:26:50.587691 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls podName:a4508991-2ff0-4497-bf57-335b69626d0c nodeName:}" failed. No retries permitted until 2026-11-01 20:26:51.087670888 +0000 UTC m=+1412.032508740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls") pod "maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" (UID: "a4508991-2ff0-4497-bf57-335b69626d0c") : secret "maas-api-e2e-migrate-46c731c3-serving-cert" not found Nov 01 20:26:50.587894 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.587814 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.588651 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.588611 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.590005 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.589984 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:50.648589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:50.648524 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5ch\" (UniqueName: \"kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:51.093615 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:51.093581 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:51.095918 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:51.095893 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\") pod \"maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:51.272442 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:51.272413 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:51.644374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:51.644343 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn"] Nov 01 20:26:51.647588 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:26:51.647547 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4508991_2ff0_4497_bf57_335b69626d0c.slice/crio-ff2a9b9ccb7422f38d5069bc509f0d355d1468c4d20b934fb067a32595001d31 WatchSource:0}: Error finding container ff2a9b9ccb7422f38d5069bc509f0d355d1468c4d20b934fb067a32595001d31: Status 404 returned error can't find the container with id ff2a9b9ccb7422f38d5069bc509f0d355d1468c4d20b934fb067a32595001d31 Nov 01 20:26:51.824315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:51.824283 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" event={"ID":"a4508991-2ff0-4497-bf57-335b69626d0c","Type":"ContainerStarted","Data":"ff2a9b9ccb7422f38d5069bc509f0d355d1468c4d20b934fb067a32595001d31"} Nov 01 20:26:52.830066 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:52.830031 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" event={"ID":"a4508991-2ff0-4497-bf57-335b69626d0c","Type":"ContainerStarted","Data":"178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955"} Nov 01 20:26:52.830433 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:52.830176 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:52.943598 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:52.943554 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" podStartSLOduration=2.090584764 podStartE2EDuration="2.943542354s" totalImagesPullingTime="852.95759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:52.943294384 +0000 UTC m=+1413.888132253" watchObservedRunningTime="2026-11-01 20:26:52.943542354 +0000 UTC m=+1413.888380225" Nov 01 20:26:58.839557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:58.839527 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:26:59.867697 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:26:59.867665 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq"] Nov 01 20:27:00.060015 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:27:00.059974 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:27:14.902391 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:14.902344 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" podUID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" containerName="main" containerID="cri-o://170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24" gracePeriod=45 Nov 01 20:27:15.030562 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.029845 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" podUID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": dial tcp 10.133.0.49:8000: connect: connection refused" Nov 01 20:27:15.031103 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:27:15.031077 5444 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb7d1c0_9a82_47f4_89a2_e03ebd9a2eda.slice/crio-170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24.scope/cpu.weight": read /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb7d1c0_9a82_47f4_89a2_e03ebd9a2eda.slice/crio-170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24.scope/cpu.weight: no such device Nov 01 20:27:15.051145 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:27:15.051114 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb7d1c0_9a82_47f4_89a2_e03ebd9a2eda.slice/crio-170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:27:15.454777 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.454746 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:27:15.533570 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533540 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm\") pod \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " Nov 01 20:27:15.533784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533607 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d\" (UniqueName: \"kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d\") pod \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " Nov 01 20:27:15.533784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533678 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir\") pod \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " Nov 01 20:27:15.533784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533715 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home\") pod \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " Nov 01 20:27:15.533784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533756 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache\") pod \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " Nov 01 20:27:15.534015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533842 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs\") pod \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\" (UID: \"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda\") " Nov 01 20:27:15.534054 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.533984 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir" pod "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" (UID: "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:27:15.534054 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.534038 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache" pod "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" (UID: "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:27:15.534222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.534200 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:15.534351 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.534221 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:15.534351 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.534293 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home" pod "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" (UID: "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:27:15.535946 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.535918 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs" pod "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" (UID: "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:27:15.536029 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.535957 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm" pod "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" (UID: "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:27:15.536069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.536024 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d" pod "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" (UID: "3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda"). InnerVolumeSpecName "kube-api-access-bs24d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:27:15.636650 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.636609 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:15.636801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.636660 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs24d\" (UniqueName: \"kubernetes.io/projected/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-kube-api-access-bs24d\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:15.636801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.636685 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:15.636801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:15.636698 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:27:16.032567 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.032537 5444 generic.go:356] "Generic (PLEG): container finished" podID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" containerID="170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24" exitCode=0 Nov 01 20:27:16.033055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.032669 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" Nov 01 20:27:16.033055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.032681 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" event={"ID":"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda","Type":"ContainerDied","Data":"170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24"} Nov 01 20:27:16.033055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.032732 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq" event={"ID":"3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda","Type":"ContainerDied","Data":"43d99f86e6f3278559bee00b167367823346d195a40bc8c2c3f554e2baf29783"} Nov 01 20:27:16.033055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.032757 5444 scope.go:122] "RemoveContainer" containerID="170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24" Nov 01 20:27:16.042065 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.042045 5444 scope.go:122] "RemoveContainer" containerID="170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24" Nov 01 20:27:16.042326 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:27:16.042305 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24\": container with ID starting with 170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24 not found: ID does not exist" containerID="170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24" Nov 01 20:27:16.042399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.042333 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24"} err="failed to get container status \"170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24\": rpc error: code = NotFound desc = could not find container \"170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24\": container with ID starting with 170865ece86116b8d82270a2ecb631959144285aa1351a7c941f3b6d394dbb24 not found: ID does not exist" Nov 01 20:27:16.196451 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.196419 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq"] Nov 01 20:27:16.199392 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:16.199370 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/rate-test-model-2cc24d-kserve-7f94ff6445-8mjqq"] Nov 01 20:27:17.158684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:27:17.158646 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" path="/var/lib/kubelet/pods/3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda/volumes" Nov 01 20:27:30.035065 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:27:30.035030 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:28:00.001979 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.001941 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7367-5c89e0"] Nov 01 20:28:00.002429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.002361 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" containerName="main" Nov 01 20:28:00.002429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.002372 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" containerName="main" Nov 01 20:28:00.002501 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.002441 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb7d1c0-9a82-47f4-89a2-e03ebd9a2eda" containerName="main" Nov 01 20:28:00.005232 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.005215 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:00.034876 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:00.034851 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:28:00.059289 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.059267 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7367-5c89e0"] Nov 01 20:28:00.106338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.106321 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wlz8j\"" Nov 01 20:28:00.118067 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.118039 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtngz\" (UniqueName: \"kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz\") pod \"test-curl-7367-5c89e0\" (UID: \"14246fbe-a768-4b70-b770-0fab56d55edf\") " pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:00.218645 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.218618 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtngz\" (UniqueName: \"kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz\") pod \"test-curl-7367-5c89e0\" (UID: \"14246fbe-a768-4b70-b770-0fab56d55edf\") " pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:00.274240 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.274214 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtngz\" (UniqueName: \"kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz\") pod \"test-curl-7367-5c89e0\" (UID: \"14246fbe-a768-4b70-b770-0fab56d55edf\") " pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:00.318449 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.318425 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:00.685422 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:00.685371 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7367-5c89e0"] Nov 01 20:28:00.688348 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:28:00.688321 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14246fbe_a768_4b70_b770_0fab56d55edf.slice/crio-45ca6a920daeec2b9f393a152ae7368842594227f48d1a2750757cdebe74fb0d WatchSource:0}: Error finding container 45ca6a920daeec2b9f393a152ae7368842594227f48d1a2750757cdebe74fb0d: Status 404 returned error can't find the container with id 45ca6a920daeec2b9f393a152ae7368842594227f48d1a2750757cdebe74fb0d Nov 01 20:28:01.303000 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.302970 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7367-5c89e0" event={"ID":"14246fbe-a768-4b70-b770-0fab56d55edf","Type":"ContainerStarted","Data":"45ca6a920daeec2b9f393a152ae7368842594227f48d1a2750757cdebe74fb0d"} Nov 01 20:28:01.734857 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.734791 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p"] Nov 01 20:28:01.739465 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.739446 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:01.840805 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.840780 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"openshift-service-ca.crt\"" Nov 01 20:28:01.841400 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.841342 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"default-dockercfg-2pqzm\"" Nov 01 20:28:01.841400 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.841385 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"kube-root-ca.crt\"" Nov 01 20:28:01.845020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.843592 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2cc24d\"/\"sub-test-model-2cc24d-kserve-self-signed-certs\"" Nov 01 20:28:01.850380 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.850353 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p"] Nov 01 20:28:01.933084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.933043 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:01.933249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.933115 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnc9h\" (UniqueName: \"kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:01.933249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.933214 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:01.933334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.933259 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:01.933435 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.933351 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:01.933494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:01.933455 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034200 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034166 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034461 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034441 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnc9h\" (UniqueName: \"kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034569 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034494 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034569 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034538 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034583 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034679 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.034947 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.034920 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.035123 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.035098 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.035225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.035185 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.037483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.037447 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.037832 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.037809 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.091503 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.091469 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnc9h\" (UniqueName: \"kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h\") pod \"sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.352954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.352877 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:02.729327 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:02.729297 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p"] Nov 01 20:28:02.732331 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:28:02.732303 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod929d1b1a_bb8d_470a_abae_a463d511a824.slice/crio-a29708d2cdcc8d7431e2aeaf90205eb9f4e9bf00d769b288f27112ffff101bec WatchSource:0}: Error finding container a29708d2cdcc8d7431e2aeaf90205eb9f4e9bf00d769b288f27112ffff101bec: Status 404 returned error can't find the container with id a29708d2cdcc8d7431e2aeaf90205eb9f4e9bf00d769b288f27112ffff101bec Nov 01 20:28:03.315995 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:03.315953 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" event={"ID":"929d1b1a-bb8d-470a-abae-a463d511a824","Type":"ContainerStarted","Data":"974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075"} Nov 01 20:28:03.316177 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:03.316006 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" event={"ID":"929d1b1a-bb8d-470a-abae-a463d511a824","Type":"ContainerStarted","Data":"a29708d2cdcc8d7431e2aeaf90205eb9f4e9bf00d769b288f27112ffff101bec"} Nov 01 20:28:03.353587 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:03.353557 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:03.429089 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:03.429039 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" podStartSLOduration=2.429023284 podStartE2EDuration="2.429023284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:03.428542569 +0000 UTC m=+1484.373380439" watchObservedRunningTime="2026-11-01 20:28:03.429023284 +0000 UTC m=+1484.373861153" Nov 01 20:28:04.322006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:04.321974 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7367-5c89e0" event={"ID":"14246fbe-a768-4b70-b770-0fab56d55edf","Type":"ContainerStarted","Data":"28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1"} Nov 01 20:28:04.433489 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:04.433443 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7367-5c89e0" podStartSLOduration=2.61362773 podStartE2EDuration="5.433428586s" totalImagesPullingTime="2.819800856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:04.431960969 +0000 UTC m=+1485.376798837" watchObservedRunningTime="2026-11-01 20:28:04.433428586 +0000 UTC m=+1485.378266454" Nov 01 20:28:05.872186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:05.872156 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7367-5c89e0"] Nov 01 20:28:05.876084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:05.876046 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7367-5c89e0"] Nov 01 20:28:06.332244 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:06.332201 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7367-5c89e0" podUID="14246fbe-a768-4b70-b770-0fab56d55edf" containerName="test-curl-7367-5c89e0" containerID="cri-o://28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1" gracePeriod=2 Nov 01 20:28:06.382820 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:06.382789 5444 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7367-5c89e0\" is forbidden: User \"system:node:ip-10-0-138-177.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-177.us-west-1.compute.internal' and this object" podUID="14246fbe-a768-4b70-b770-0fab56d55edf" pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:08.604924 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:08.604894 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:08.705167 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:08.705143 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz\" (UniqueName: \"kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz\") pod \"14246fbe-a768-4b70-b770-0fab56d55edf\" (UID: \"14246fbe-a768-4b70-b770-0fab56d55edf\") " Nov 01 20:28:08.707031 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:08.707009 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz" pod "14246fbe-a768-4b70-b770-0fab56d55edf" (UID: "14246fbe-a768-4b70-b770-0fab56d55edf"). InnerVolumeSpecName "kube-api-access-mtngz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:08.805889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:08.805868 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtngz\" (UniqueName: \"kubernetes.io/projected/14246fbe-a768-4b70-b770-0fab56d55edf-kube-api-access-mtngz\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:09.159662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:09.159605 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14246fbe-a768-4b70-b770-0fab56d55edf" path="/var/lib/kubelet/pods/14246fbe-a768-4b70-b770-0fab56d55edf/volumes" Nov 01 20:28:09.347444 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:09.347417 5444 generic.go:356] "Generic (PLEG): container finished" podID="14246fbe-a768-4b70-b770-0fab56d55edf" containerID="28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1" exitCode=137 Nov 01 20:28:09.347542 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:09.347452 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-5c89e0" Nov 01 20:28:09.347542 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:09.347487 5444 scope.go:122] "RemoveContainer" containerID="28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1" Nov 01 20:28:09.355867 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:09.355850 5444 scope.go:122] "RemoveContainer" containerID="28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1" Nov 01 20:28:09.356124 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:09.356105 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1\": container with ID starting with 28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1 not found: ID does not exist" containerID="28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1" Nov 01 20:28:09.356185 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:09.356131 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1"} err="failed to get container status \"28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1\": rpc error: code = NotFound desc = could not find container \"28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1\": container with ID starting with 28de29bee1200ae8713ccc869148543b06bbc856aca09d91cbdfb8cb8a9096b1 not found: ID does not exist" Nov 01 20:28:12.353209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:12.353165 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:12.369241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:12.369210 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:12.385945 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:12.385920 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:28:17.065991 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.065949 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697"] Nov 01 20:28:17.066449 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.066432 5444 cpu_manager.go:414] "containerMap: removing container" podUID="14246fbe-a768-4b70-b770-0fab56d55edf" containerName="test-curl-7367-5c89e0" Nov 01 20:28:17.066498 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.066451 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14246fbe-a768-4b70-b770-0fab56d55edf" containerName="test-curl-7367-5c89e0" Nov 01 20:28:17.066533 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.066521 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="14246fbe-a768-4b70-b770-0fab56d55edf" containerName="test-curl-7367-5c89e0" Nov 01 20:28:17.070980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.070964 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.112877 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.112845 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frclx\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.112980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.112909 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.112980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.112972 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.113050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.113007 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.113050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.113024 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.113050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.113042 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.113153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.113120 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.113206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.113150 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.113206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.113167 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.174371 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.174342 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697"] Nov 01 20:28:17.213852 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.213831 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.213987 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.213858 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.213987 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.213881 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.213987 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.213914 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.213987 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.213945 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214236 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214065 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frclx\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214236 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214146 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214361 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214278 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214361 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214349 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214361 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214435 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214793 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214772 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.214965 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.214812 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.215357 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.215336 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.216562 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.216543 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.217405 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.217383 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.273664 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.273625 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frclx\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.273747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.273705 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.384128 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.384076 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:17.771674 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.769463 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn"] Nov 01 20:28:17.771674 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.769789 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" podUID="a4508991-2ff0-4497-bf57-335b69626d0c" containerName="maas-api" containerID="cri-o://178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955" gracePeriod=30 Nov 01 20:28:17.817609 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.817577 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697"] Nov 01 20:28:17.854526 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:28:17.854501 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c1328b_d892_4179_898b_22ea94729d04.slice/crio-1762853e37efed3cbed7b0997526ced4c63d95e8c98fd17109882f40f4d57adc WatchSource:0}: Error finding container 1762853e37efed3cbed7b0997526ced4c63d95e8c98fd17109882f40f4d57adc: Status 404 returned error can't find the container with id 1762853e37efed3cbed7b0997526ced4c63d95e8c98fd17109882f40f4d57adc Nov 01 20:28:17.878983 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.878938 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:28:17.879092 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.879024 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:28:17.879092 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:17.879070 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:28:18.120693 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.120403 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:28:18.138342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.135295 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch\" (UniqueName: \"kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch\") pod \"a4508991-2ff0-4497-bf57-335b69626d0c\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " Nov 01 20:28:18.138342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.135349 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\") pod \"a4508991-2ff0-4497-bf57-335b69626d0c\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " Nov 01 20:28:18.138342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.135367 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs\") pod \"a4508991-2ff0-4497-bf57-335b69626d0c\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " Nov 01 20:28:18.138342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.135821 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca\") pod \"a4508991-2ff0-4497-bf57-335b69626d0c\" (UID: \"a4508991-2ff0-4497-bf57-335b69626d0c\") " Nov 01 20:28:18.138342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.136616 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca" pod "a4508991-2ff0-4497-bf57-335b69626d0c" (UID: "a4508991-2ff0-4497-bf57-335b69626d0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:18.139859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.139821 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls" pod "a4508991-2ff0-4497-bf57-335b69626d0c" (UID: "a4508991-2ff0-4497-bf57-335b69626d0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:28:18.140084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.140058 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch" pod "a4508991-2ff0-4497-bf57-335b69626d0c" (UID: "a4508991-2ff0-4497-bf57-335b69626d0c"). InnerVolumeSpecName "kube-api-access-zw5ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:18.140934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.140426 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs" pod "a4508991-2ff0-4497-bf57-335b69626d0c" (UID: "a4508991-2ff0-4497-bf57-335b69626d0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:28:18.237873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.237838 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5ch\" (UniqueName: \"kubernetes.io/projected/a4508991-2ff0-4497-bf57-335b69626d0c-kube-api-access-zw5ch\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:18.238018 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.237875 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:18.238018 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.237891 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4508991-2ff0-4497-bf57-335b69626d0c-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:18.238018 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.237904 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4508991-2ff0-4497-bf57-335b69626d0c-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:28:18.411180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.411095 5444 generic.go:356] "Generic (PLEG): container finished" podID="a4508991-2ff0-4497-bf57-335b69626d0c" containerID="178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955" exitCode=0 Nov 01 20:28:18.411180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.411141 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" Nov 01 20:28:18.411408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.411186 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" event={"ID":"a4508991-2ff0-4497-bf57-335b69626d0c","Type":"ContainerDied","Data":"178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955"} Nov 01 20:28:18.411408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.411241 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn" event={"ID":"a4508991-2ff0-4497-bf57-335b69626d0c","Type":"ContainerDied","Data":"ff2a9b9ccb7422f38d5069bc509f0d355d1468c4d20b934fb067a32595001d31"} Nov 01 20:28:18.411408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.411264 5444 scope.go:122] "RemoveContainer" containerID="178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955" Nov 01 20:28:18.412963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.412934 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" event={"ID":"a9c1328b-d892-4179-898b-22ea94729d04","Type":"ContainerStarted","Data":"a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb"} Nov 01 20:28:18.413105 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.412983 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" event={"ID":"a9c1328b-d892-4179-898b-22ea94729d04","Type":"ContainerStarted","Data":"1762853e37efed3cbed7b0997526ced4c63d95e8c98fd17109882f40f4d57adc"} Nov 01 20:28:18.423365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.423220 5444 scope.go:122] "RemoveContainer" containerID="178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955" Nov 01 20:28:18.423542 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:18.423525 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955\": container with ID starting with 178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955 not found: ID does not exist" containerID="178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955" Nov 01 20:28:18.423592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.423557 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955"} err="failed to get container status \"178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955\": rpc error: code = NotFound desc = could not find container \"178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955\": container with ID starting with 178b0794abefb658cb66b1a16897554299c15028a01a75554350d155b41ac955 not found: ID does not exist" Nov 01 20:28:18.590661 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.590425 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn"] Nov 01 20:28:18.614657 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.613798 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-46c731c3-b8fd757f8-sjztn"] Nov 01 20:28:18.699813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:18.699689 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podStartSLOduration=1.699673862 podStartE2EDuration="1.699673862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:18.699389807 +0000 UTC m=+1499.644227675" watchObservedRunningTime="2026-11-01 20:28:18.699673862 +0000 UTC m=+1499.644511727" Nov 01 20:28:19.201569 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.201537 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4508991-2ff0-4497-bf57-335b69626d0c" path="/var/lib/kubelet/pods/a4508991-2ff0-4497-bf57-335b69626d0c/volumes" Nov 01 20:28:19.384427 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.384389 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:19.385714 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.385690 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:28:19.385825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.385735 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 01 20:28:19.470708 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.470581 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz"] Nov 01 20:28:19.471678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.471212 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a4508991-2ff0-4497-bf57-335b69626d0c" containerName="maas-api" Nov 01 20:28:19.471678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.471235 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4508991-2ff0-4497-bf57-335b69626d0c" containerName="maas-api" Nov 01 20:28:19.471678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.471364 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4508991-2ff0-4497-bf57-335b69626d0c" containerName="maas-api" Nov 01 20:28:19.477089 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.477067 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.480526 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.480494 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.480682 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.480617 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.480809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.480721 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.480809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.480753 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcnn\" (UniqueName: \"kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.532012 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.531993 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-202d23-serving-cert\"" Nov 01 20:28:19.534742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.534711 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz"] Nov 01 20:28:19.581509 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.581475 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.581695 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.581529 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.581695 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.581560 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.581695 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.581582 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcnn\" (UniqueName: \"kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.581881 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:19.581733 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-202d23-serving-cert: secret "maas-api-e2e-autores-202d23-serving-cert" not found Nov 01 20:28:19.581881 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:19.581808 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls podName:9dfdf434-14d0-4d38-a9c2-97473d349459 nodeName:}" failed. No retries permitted until 2026-11-01 20:28:20.08178844 +0000 UTC m=+1501.026626290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls") pod "maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" (UID: "9dfdf434-14d0-4d38-a9c2-97473d349459") : secret "maas-api-e2e-autores-202d23-serving-cert" not found Nov 01 20:28:19.582448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.582424 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.584094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.584074 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:19.639870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:19.639846 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcnn\" (UniqueName: \"kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:20.086349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:20.086314 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:20.088785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:20.088763 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\") pod \"maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:20.090484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:20.090465 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:20.385235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:20.385164 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:28:20.385235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:20.385216 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 01 20:28:20.615897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:20.615769 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz"] Nov 01 20:28:20.618671 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:28:20.618632 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfdf434_14d0_4d38_a9c2_97473d349459.slice/crio-69e10c8a0f400964852238c730af1c7277223ec172fdc293419826a8ed4caa4e WatchSource:0}: Error finding container 69e10c8a0f400964852238c730af1c7277223ec172fdc293419826a8ed4caa4e: Status 404 returned error can't find the container with id 69e10c8a0f400964852238c730af1c7277223ec172fdc293419826a8ed4caa4e Nov 01 20:28:21.389658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:21.388691 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:28:21.389658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:21.388749 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 01 20:28:21.430819 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:21.430781 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" event={"ID":"9dfdf434-14d0-4d38-a9c2-97473d349459","Type":"ContainerStarted","Data":"69e10c8a0f400964852238c730af1c7277223ec172fdc293419826a8ed4caa4e"} Nov 01 20:28:22.384394 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:22.384358 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:28:22.384561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:22.384435 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 01 20:28:22.477968 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:22.477927 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" event={"ID":"9dfdf434-14d0-4d38-a9c2-97473d349459","Type":"ContainerStarted","Data":"9406f160fa934a43b112582e04697d953e02010541e463532415898f8983bf35"} Nov 01 20:28:22.604692 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:22.604620 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" podStartSLOduration=2.605812889 podStartE2EDuration="3.604598712s" totalImagesPullingTime="998.785823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:22.603959012 +0000 UTC m=+1503.548796880" watchObservedRunningTime="2026-11-01 20:28:22.604598712 +0000 UTC m=+1503.549436580" Nov 01 20:28:23.384692 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:23.384630 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:28:23.384891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:23.384713 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 01 20:28:23.486122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:23.486098 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:24.384545 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:24.384506 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 01 20:28:24.384728 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:24.384576 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 01 20:28:25.389430 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:25.389398 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:25.389866 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:25.389743 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:25.390489 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:25.390468 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:28:29.497157 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:28:29.497130 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:28:30.036767 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:28:30.036731 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:29:00.034192 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:29:00.034113 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:29:03.947989 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:03.947951 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl"] Nov 01 20:29:03.951814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:03.951792 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:03.981539 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:03.981513 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9lvh\" (UniqueName: \"kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:03.981659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:03.981553 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:03.981659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:03.981572 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:03.981659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:03.981618 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.008266 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.008243 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl"] Nov 01 20:29:04.021776 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.021755 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7387069c-serving-cert\"" Nov 01 20:29:04.082617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.082585 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.082617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.082616 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.082844 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.082651 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.082844 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:29:04.082750 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-serving-cert: secret "maas-api-e2e-ait-7387069c-serving-cert" not found Nov 01 20:29:04.082844 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.082801 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9lvh\" (UniqueName: \"kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.082844 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:29:04.082813 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls podName:77ccad7f-a128-407b-9c4c-3fddef93297d nodeName:}" failed. No retries permitted until 2026-11-01 20:29:04.582799277 +0000 UTC m=+1545.527637132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls") pod "maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" (UID: "77ccad7f-a128-407b-9c4c-3fddef93297d") : secret "maas-api-e2e-ait-7387069c-serving-cert" not found Nov 01 20:29:04.083606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.083588 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.085746 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.085721 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.140940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.140917 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9lvh\" (UniqueName: \"kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.594524 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.594490 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.596860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.596836 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\") pod \"maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:04.864609 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:04.864547 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:05.235268 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:05.235245 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl"] Nov 01 20:29:05.238058 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:29:05.238025 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ccad7f_a128_407b_9c4c_3fddef93297d.slice/crio-ab20b4b9d0d5da3ead600a8a2dc9cbb07576edc8f145f424648527a4303939a1 WatchSource:0}: Error finding container ab20b4b9d0d5da3ead600a8a2dc9cbb07576edc8f145f424648527a4303939a1: Status 404 returned error can't find the container with id ab20b4b9d0d5da3ead600a8a2dc9cbb07576edc8f145f424648527a4303939a1 Nov 01 20:29:05.746289 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:05.746254 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" event={"ID":"77ccad7f-a128-407b-9c4c-3fddef93297d","Type":"ContainerStarted","Data":"ab20b4b9d0d5da3ead600a8a2dc9cbb07576edc8f145f424648527a4303939a1"} Nov 01 20:29:06.753149 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:06.753098 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" event={"ID":"77ccad7f-a128-407b-9c4c-3fddef93297d","Type":"ContainerStarted","Data":"f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64"} Nov 01 20:29:06.753530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:06.753186 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:06.867981 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:06.867936 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" podStartSLOduration=3.025338378 podStartE2EDuration="3.867923373s" totalImagesPullingTime="842.584995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:06.865944118 +0000 UTC m=+1547.810781986" watchObservedRunningTime="2026-11-01 20:29:06.867923373 +0000 UTC m=+1547.812761240" Nov 01 20:29:12.764284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:12.764251 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:29:27.855937 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:27.855615 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc"] Nov 01 20:29:27.855937 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:27.855883 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" podUID="ab46a205-b355-4ad8-ad32-e61386b2014a" containerName="maas-api" containerID="cri-o://b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a" gracePeriod=30 Nov 01 20:29:28.173670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.173629 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:29:28.350295 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.350266 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\") pod \"ab46a205-b355-4ad8-ad32-e61386b2014a\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " Nov 01 20:29:28.350447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.350385 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw\" (UniqueName: \"kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw\") pod \"ab46a205-b355-4ad8-ad32-e61386b2014a\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " Nov 01 20:29:28.350447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.350434 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca\") pod \"ab46a205-b355-4ad8-ad32-e61386b2014a\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " Nov 01 20:29:28.350540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.350456 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs\") pod \"ab46a205-b355-4ad8-ad32-e61386b2014a\" (UID: \"ab46a205-b355-4ad8-ad32-e61386b2014a\") " Nov 01 20:29:28.350891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.350863 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca" pod "ab46a205-b355-4ad8-ad32-e61386b2014a" (UID: "ab46a205-b355-4ad8-ad32-e61386b2014a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:29:28.352371 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.352346 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls" pod "ab46a205-b355-4ad8-ad32-e61386b2014a" (UID: "ab46a205-b355-4ad8-ad32-e61386b2014a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:29:28.352498 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.352421 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw" pod "ab46a205-b355-4ad8-ad32-e61386b2014a" (UID: "ab46a205-b355-4ad8-ad32-e61386b2014a"). InnerVolumeSpecName "kube-api-access-vmrsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:29:28.352498 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.352445 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs" pod "ab46a205-b355-4ad8-ad32-e61386b2014a" (UID: "ab46a205-b355-4ad8-ad32-e61386b2014a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:29:28.452457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.452373 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrsw\" (UniqueName: \"kubernetes.io/projected/ab46a205-b355-4ad8-ad32-e61386b2014a-kube-api-access-vmrsw\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:29:28.452457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.452407 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab46a205-b355-4ad8-ad32-e61386b2014a-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:29:28.452457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.452419 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:29:28.452457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.452431 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab46a205-b355-4ad8-ad32-e61386b2014a-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:29:28.869398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.869363 5444 generic.go:356] "Generic (PLEG): container finished" podID="ab46a205-b355-4ad8-ad32-e61386b2014a" containerID="b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a" exitCode=0 Nov 01 20:29:28.869903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.869431 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" Nov 01 20:29:28.869903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.869517 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" event={"ID":"ab46a205-b355-4ad8-ad32-e61386b2014a","Type":"ContainerDied","Data":"b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a"} Nov 01 20:29:28.869903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.869651 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc" event={"ID":"ab46a205-b355-4ad8-ad32-e61386b2014a","Type":"ContainerDied","Data":"d87779f42e6e6af695560b5b6b6f41abc4078b0537f71dac0574f0c6f3a86d08"} Nov 01 20:29:28.869903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.869691 5444 scope.go:122] "RemoveContainer" containerID="b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a" Nov 01 20:29:28.880146 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.880129 5444 scope.go:122] "RemoveContainer" containerID="b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a" Nov 01 20:29:28.880406 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:29:28.880386 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a\": container with ID starting with b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a not found: ID does not exist" containerID="b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a" Nov 01 20:29:28.880444 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:28.880415 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a"} err="failed to get container status \"b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a\": rpc error: code = NotFound desc = could not find container \"b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a\": container with ID starting with b3e0bd38ae1b8413d83d763686623109386dcce14c3db47bcd1c9c70bcd1597a not found: ID does not exist" Nov 01 20:29:29.030446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:29.030425 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc"] Nov 01 20:29:29.034267 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:29.034247 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-6mbtc"] Nov 01 20:29:29.157419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:29:29.157353 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab46a205-b355-4ad8-ad32-e61386b2014a" path="/var/lib/kubelet/pods/ab46a205-b355-4ad8-ad32-e61386b2014a/volumes" Nov 01 20:29:30.034115 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:29:30.034083 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:30:00.034667 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:00.034612 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:30:00.187589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.187560 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt"] Nov 01 20:30:00.187985 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.187973 5444 cpu_manager.go:414] "containerMap: removing container" podUID="ab46a205-b355-4ad8-ad32-e61386b2014a" containerName="maas-api" Nov 01 20:30:00.188033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.187988 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab46a205-b355-4ad8-ad32-e61386b2014a" containerName="maas-api" Nov 01 20:30:00.188082 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.188074 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab46a205-b355-4ad8-ad32-e61386b2014a" containerName="maas-api" Nov 01 20:30:00.191193 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.191178 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.263715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.263687 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdn4\" (UniqueName: \"kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4\") pod \"maas-api-key-cleanup-29892750-97xqt\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.263962 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.263940 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892750-97xqt\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.354046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.353991 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 20:30:00.365420 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.365397 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdn4\" (UniqueName: \"kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4\") pod \"maas-api-key-cleanup-29892750-97xqt\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.365495 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.365446 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892750-97xqt\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.366042 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.366027 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892750-97xqt\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.386451 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.386425 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt"] Nov 01 20:30:00.421705 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.421687 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdn4\" (UniqueName: \"kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4\") pod \"maas-api-key-cleanup-29892750-97xqt\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.503573 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.503549 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:00.871201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:00.871165 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt"] Nov 01 20:30:00.875427 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:30:00.875394 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b053fdd_bd07_4574_881e_26b25e3e193c.slice/crio-caeb22c98ccdd579a47abc58b1572f98b6f65ec2be2435ae579628dcb5e0eaca WatchSource:0}: Error finding container caeb22c98ccdd579a47abc58b1572f98b6f65ec2be2435ae579628dcb5e0eaca: Status 404 returned error can't find the container with id caeb22c98ccdd579a47abc58b1572f98b6f65ec2be2435ae579628dcb5e0eaca Nov 01 20:30:01.062306 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:01.062271 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" event={"ID":"0b053fdd-bd07-4574-881e-26b25e3e193c","Type":"ContainerStarted","Data":"caeb22c98ccdd579a47abc58b1572f98b6f65ec2be2435ae579628dcb5e0eaca"} Nov 01 20:30:02.068111 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:02.068078 5444 generic.go:356] "Generic (PLEG): container finished" podID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerID="d8c6d351ebeebccfc1d9bd19ffa790d53e397518670aabd7c5e062e5e3029b16" exitCode=60 Nov 01 20:30:02.068485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:02.068163 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" event={"ID":"0b053fdd-bd07-4574-881e-26b25e3e193c","Type":"ContainerDied","Data":"d8c6d351ebeebccfc1d9bd19ffa790d53e397518670aabd7c5e062e5e3029b16"} Nov 01 20:30:02.068485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:02.068325 5444 scope.go:122] "RemoveContainer" containerID="d8c6d351ebeebccfc1d9bd19ffa790d53e397518670aabd7c5e062e5e3029b16" Nov 01 20:30:03.076525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:03.076492 5444 generic.go:356] "Generic (PLEG): container finished" podID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerID="4df4b0092250c76000d3ddd12dbb63a59291ed9767eafb786e349992f4da2553" exitCode=60 Nov 01 20:30:03.076916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:03.076562 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" event={"ID":"0b053fdd-bd07-4574-881e-26b25e3e193c","Type":"ContainerDied","Data":"4df4b0092250c76000d3ddd12dbb63a59291ed9767eafb786e349992f4da2553"} Nov 01 20:30:03.076916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:03.076621 5444 scope.go:122] "RemoveContainer" containerID="d8c6d351ebeebccfc1d9bd19ffa790d53e397518670aabd7c5e062e5e3029b16" Nov 01 20:30:03.077112 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:03.077092 5444 scope.go:122] "RemoveContainer" containerID="4df4b0092250c76000d3ddd12dbb63a59291ed9767eafb786e349992f4da2553" Nov 01 20:30:03.077391 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:03.077362 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892750-97xqt_odh-ai-gateway-infra(0b053fdd-bd07-4574-881e-26b25e3e193c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" Nov 01 20:30:04.083374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:04.083347 5444 scope.go:122] "RemoveContainer" containerID="4df4b0092250c76000d3ddd12dbb63a59291ed9767eafb786e349992f4da2553" Nov 01 20:30:04.083906 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:04.083661 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892750-97xqt_odh-ai-gateway-infra(0b053fdd-bd07-4574-881e-26b25e3e193c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" Nov 01 20:30:14.152408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.152380 5444 scope.go:122] "RemoveContainer" containerID="4df4b0092250c76000d3ddd12dbb63a59291ed9767eafb786e349992f4da2553" Nov 01 20:30:14.199696 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.199673 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x"] Nov 01 20:30:14.204319 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.204304 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.258688 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.258665 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71335882-serving-cert\"" Nov 01 20:30:14.261008 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.260983 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x"] Nov 01 20:30:14.337421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.337397 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.337536 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.337447 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.337581 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.337566 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.337661 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.337626 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxgr\" (UniqueName: \"kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.438825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.438757 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.438940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.438846 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxgr\" (UniqueName: \"kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.438940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.438891 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.439059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.438956 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.439132 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:14.439110 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-71335882-serving-cert: secret "maas-api-e2e-mt-71335882-serving-cert" not found Nov 01 20:30:14.439194 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:14.439184 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls podName:3a6dc35b-ffc2-4e29-800d-f40a62751713 nodeName:}" failed. No retries permitted until 2026-11-01 20:30:14.939165661 +0000 UTC m=+1615.884003529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls") pod "maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" (UID: "3a6dc35b-ffc2-4e29-800d-f40a62751713") : secret "maas-api-e2e-mt-71335882-serving-cert" not found Nov 01 20:30:14.439417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.439397 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.441111 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.441093 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.521364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.521335 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxgr\" (UniqueName: \"kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.943605 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.943571 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:14.946128 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:14.946100 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\") pod \"maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:15.116293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:15.116263 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:15.140045 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:15.140014 5444 generic.go:356] "Generic (PLEG): container finished" podID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerID="23e798541e45512b71adb0633c7e7375fb4291b59ebb7e720d6c71d5c8f33fc9" exitCode=60 Nov 01 20:30:15.140192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:15.140082 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" event={"ID":"0b053fdd-bd07-4574-881e-26b25e3e193c","Type":"ContainerDied","Data":"23e798541e45512b71adb0633c7e7375fb4291b59ebb7e720d6c71d5c8f33fc9"} Nov 01 20:30:15.140192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:15.140140 5444 scope.go:122] "RemoveContainer" containerID="4df4b0092250c76000d3ddd12dbb63a59291ed9767eafb786e349992f4da2553" Nov 01 20:30:15.140471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:15.140452 5444 scope.go:122] "RemoveContainer" containerID="23e798541e45512b71adb0633c7e7375fb4291b59ebb7e720d6c71d5c8f33fc9" Nov 01 20:30:15.140744 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:15.140722 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892750-97xqt_odh-ai-gateway-infra(0b053fdd-bd07-4574-881e-26b25e3e193c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" Nov 01 20:30:15.542633 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:15.542499 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x"] Nov 01 20:30:15.545401 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:30:15.545376 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6dc35b_ffc2_4e29_800d_f40a62751713.slice/crio-e03654409835a62892df855c03bc4766b277702e32f37c8813d7823e1ccc6282 WatchSource:0}: Error finding container e03654409835a62892df855c03bc4766b277702e32f37c8813d7823e1ccc6282: Status 404 returned error can't find the container with id e03654409835a62892df855c03bc4766b277702e32f37c8813d7823e1ccc6282 Nov 01 20:30:16.146552 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.146517 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" event={"ID":"3a6dc35b-ffc2-4e29-800d-f40a62751713","Type":"ContainerStarted","Data":"e03654409835a62892df855c03bc4766b277702e32f37c8813d7823e1ccc6282"} Nov 01 20:30:16.263800 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.263762 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt"] Nov 01 20:30:16.464099 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.464066 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:16.567293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.567268 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4\" (UniqueName: \"kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4\") pod \"0b053fdd-bd07-4574-881e-26b25e3e193c\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " Nov 01 20:30:16.567678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.567413 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca\") pod \"0b053fdd-bd07-4574-881e-26b25e3e193c\" (UID: \"0b053fdd-bd07-4574-881e-26b25e3e193c\") " Nov 01 20:30:16.567784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.567759 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca" pod "0b053fdd-bd07-4574-881e-26b25e3e193c" (UID: "0b053fdd-bd07-4574-881e-26b25e3e193c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:16.569185 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.569164 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4" pod "0b053fdd-bd07-4574-881e-26b25e3e193c" (UID: "0b053fdd-bd07-4574-881e-26b25e3e193c"). InnerVolumeSpecName "kube-api-access-ccdn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:16.668926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.668892 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b053fdd-bd07-4574-881e-26b25e3e193c-maas-api-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:16.668926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:16.668920 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdn4\" (UniqueName: \"kubernetes.io/projected/0b053fdd-bd07-4574-881e-26b25e3e193c-kube-api-access-ccdn4\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:17.155040 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.155012 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" Nov 01 20:30:17.157624 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.157595 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt" event={"ID":"0b053fdd-bd07-4574-881e-26b25e3e193c","Type":"ContainerDied","Data":"caeb22c98ccdd579a47abc58b1572f98b6f65ec2be2435ae579628dcb5e0eaca"} Nov 01 20:30:17.157772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.157657 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:17.157772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.157682 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" event={"ID":"3a6dc35b-ffc2-4e29-800d-f40a62751713","Type":"ContainerStarted","Data":"348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995"} Nov 01 20:30:17.157772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.157683 5444 scope.go:122] "RemoveContainer" containerID="23e798541e45512b71adb0633c7e7375fb4291b59ebb7e720d6c71d5c8f33fc9" Nov 01 20:30:17.333360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.333308 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" podStartSLOduration=2.400515977 podStartE2EDuration="3.333293674s" totalImagesPullingTime="932.777697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:17.331397591 +0000 UTC m=+1618.276235459" watchObservedRunningTime="2026-11-01 20:30:17.333293674 +0000 UTC m=+1618.278131543" Nov 01 20:30:17.445408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.445314 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt"] Nov 01 20:30:17.461095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:17.453889 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-97xqt"] Nov 01 20:30:19.161729 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:19.161691 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" path="/var/lib/kubelet/pods/0b053fdd-bd07-4574-881e-26b25e3e193c/volumes" Nov 01 20:30:23.167625 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:23.167599 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:30:30.037464 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:30.037425 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:30:37.525102 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.525069 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl"] Nov 01 20:30:37.525676 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.525275 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" podUID="77ccad7f-a128-407b-9c4c-3fddef93297d" containerName="maas-api" containerID="cri-o://f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64" gracePeriod=30 Nov 01 20:30:37.872516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.872491 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:30:37.997998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.997965 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca\") pod \"77ccad7f-a128-407b-9c4c-3fddef93297d\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " Nov 01 20:30:37.998186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.998069 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\") pod \"77ccad7f-a128-407b-9c4c-3fddef93297d\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " Nov 01 20:30:37.998186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.998127 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs\") pod \"77ccad7f-a128-407b-9c4c-3fddef93297d\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " Nov 01 20:30:37.998186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.998180 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh\" (UniqueName: \"kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh\") pod \"77ccad7f-a128-407b-9c4c-3fddef93297d\" (UID: \"77ccad7f-a128-407b-9c4c-3fddef93297d\") " Nov 01 20:30:37.998535 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.998320 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca" pod "77ccad7f-a128-407b-9c4c-3fddef93297d" (UID: "77ccad7f-a128-407b-9c4c-3fddef93297d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:37.998993 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:37.998746 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77ccad7f-a128-407b-9c4c-3fddef93297d-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.000932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.000906 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls" pod "77ccad7f-a128-407b-9c4c-3fddef93297d" (UID: "77ccad7f-a128-407b-9c4c-3fddef93297d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:38.001617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.001592 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs" pod "77ccad7f-a128-407b-9c4c-3fddef93297d" (UID: "77ccad7f-a128-407b-9c4c-3fddef93297d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:38.001926 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.001905 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh" pod "77ccad7f-a128-407b-9c4c-3fddef93297d" (UID: "77ccad7f-a128-407b-9c4c-3fddef93297d"). InnerVolumeSpecName "kube-api-access-m9lvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:38.100098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.100044 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.100098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.100066 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77ccad7f-a128-407b-9c4c-3fddef93297d-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.100098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.100076 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9lvh\" (UniqueName: \"kubernetes.io/projected/77ccad7f-a128-407b-9c4c-3fddef93297d-kube-api-access-m9lvh\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.256503 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.256468 5444 generic.go:356] "Generic (PLEG): container finished" podID="77ccad7f-a128-407b-9c4c-3fddef93297d" containerID="f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64" exitCode=0 Nov 01 20:30:38.256666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.256516 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" Nov 01 20:30:38.256666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.256556 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" event={"ID":"77ccad7f-a128-407b-9c4c-3fddef93297d","Type":"ContainerDied","Data":"f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64"} Nov 01 20:30:38.256666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.256611 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" event={"ID":"77ccad7f-a128-407b-9c4c-3fddef93297d","Type":"ContainerDied","Data":"ab20b4b9d0d5da3ead600a8a2dc9cbb07576edc8f145f424648527a4303939a1"} Nov 01 20:30:38.256666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.256663 5444 scope.go:122] "RemoveContainer" containerID="f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64" Nov 01 20:30:38.265382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.265366 5444 scope.go:122] "RemoveContainer" containerID="f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64" Nov 01 20:30:38.265650 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:30:38.265622 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64\": container with ID starting with f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64 not found: ID does not exist" containerID="f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64" Nov 01 20:30:38.265701 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.265660 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64"} err="failed to get container status \"f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64\": rpc error: code = NotFound desc = could not find container \"f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64\": container with ID starting with f1ccfbf1c7f7e430d672e246d66013b713c63a51fc8f83dd83a02bb7602f3c64 not found: ID does not exist" Nov 01 20:30:38.417885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.417826 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl"] Nov 01 20:30:38.466993 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:38.466966 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl"] Nov 01 20:30:39.158140 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:39.158111 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ccad7f-a128-407b-9c4c-3fddef93297d" path="/var/lib/kubelet/pods/77ccad7f-a128-407b-9c4c-3fddef93297d/volumes" Nov 01 20:30:40.775054 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:40.775017 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7387069c-5d5b9b6554-cn2gl" podUID="77ccad7f-a128-407b-9c4c-3fddef93297d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510412 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql"] Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510864 5444 cpu_manager.go:414] "containerMap: removing container" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510877 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510903 5444 cpu_manager.go:414] "containerMap: removing container" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510909 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510920 5444 cpu_manager.go:414] "containerMap: removing container" podUID="77ccad7f-a128-407b-9c4c-3fddef93297d" containerName="maas-api" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.510926 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ccad7f-a128-407b-9c4c-3fddef93297d" containerName="maas-api" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.511011 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.511021 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:30:48.512666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.511030 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ccad7f-a128-407b-9c4c-3fddef93297d" containerName="maas-api" Nov 01 20:30:48.515756 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.515738 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583003 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583220 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583043 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583220 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583137 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583220 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583199 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583378 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583240 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583378 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583296 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzs6k\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583455 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583410 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583497 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583468 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.583586 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.583570 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.622473 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.622421 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71335882-openshift-default-dockercfg-f4t24\"" Nov 01 20:30:48.624781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.624705 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql"] Nov 01 20:30:48.788544 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.788352 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.794992 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.794961 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797392 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797544 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797529 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797568 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797689 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797620 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzs6k\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797750 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797700 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797807 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797880 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797849 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.797947 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.797931 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.798966 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.798945 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.799112 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.790028 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.799925 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.799885 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.800339 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.800310 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.807664 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.807628 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.808232 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.808212 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.866457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.866428 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:48.878026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:48.877967 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzs6k\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:49.135010 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:49.134919 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:49.613793 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:49.613772 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql"] Nov 01 20:30:49.616148 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:30:49.616117 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0a1063_881c_489f_9b0d_b3f3a696ad46.slice/crio-228923dbc4bae231b2c52ee668836d74341109e91b50837335dfad2817db98d8 WatchSource:0}: Error finding container 228923dbc4bae231b2c52ee668836d74341109e91b50837335dfad2817db98d8: Status 404 returned error can't find the container with id 228923dbc4bae231b2c52ee668836d74341109e91b50837335dfad2817db98d8 Nov 01 20:30:49.618394 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:49.618362 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:30:49.618456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:49.618426 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:30:49.618491 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:49.618457 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:30:50.329615 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:50.329579 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" event={"ID":"2a0a1063-881c-489f-9b0d-b3f3a696ad46","Type":"ContainerStarted","Data":"2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9"} Nov 01 20:30:50.329803 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:50.329655 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" event={"ID":"2a0a1063-881c-489f-9b0d-b3f3a696ad46","Type":"ContainerStarted","Data":"228923dbc4bae231b2c52ee668836d74341109e91b50837335dfad2817db98d8"} Nov 01 20:30:51.135503 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:51.135471 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:51.137122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:51.137098 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:30:51.137242 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:51.137145 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:30:52.135913 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:52.135878 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:30:52.136325 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:52.135958 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:30:53.136193 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:53.136160 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:30:53.136559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:53.136215 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:30:54.135847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:54.135812 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:30:54.136026 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:54.135864 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:30:55.137757 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:55.137717 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 20:30:55.138381 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:55.138345 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 20:30:56.240477 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:56.240440 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:56.240994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:56.240966 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:56.241764 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:56.241738 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:30:56.454459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:30:56.454400 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podStartSLOduration=8.454384709 podStartE2EDuration="8.454384709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:50.542892437 +0000 UTC m=+1651.487730301" watchObservedRunningTime="2026-11-01 20:30:56.454384709 +0000 UTC m=+1657.399222579" Nov 01 20:31:00.035001 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:00.034971 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:31:11.587254 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.587218 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7367-d6cf8b"] Nov 01 20:31:11.587886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.587867 5444 cpu_manager.go:414] "containerMap: removing container" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:31:11.587949 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.587888 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:31:11.588028 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.588017 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b053fdd-bd07-4574-881e-26b25e3e193c" containerName="cleanup" Nov 01 20:31:11.591436 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.591415 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:11.645740 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.645713 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7367-d6cf8b"] Nov 01 20:31:11.692715 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.692691 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wlz8j\"" Nov 01 20:31:11.706172 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.706141 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrgz\" (UniqueName: \"kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz\") pod \"test-curl-7367-d6cf8b\" (UID: \"68d66926-3d5a-4ef6-bb19-80762128b240\") " pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:11.807142 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.807117 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrgz\" (UniqueName: \"kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz\") pod \"test-curl-7367-d6cf8b\" (UID: \"68d66926-3d5a-4ef6-bb19-80762128b240\") " pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:11.863830 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.863782 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrgz\" (UniqueName: \"kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz\") pod \"test-curl-7367-d6cf8b\" (UID: \"68d66926-3d5a-4ef6-bb19-80762128b240\") " pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:11.904472 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:11.904452 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:12.269957 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:12.269934 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7367-d6cf8b"] Nov 01 20:31:12.272615 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:31:12.272587 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68d66926_3d5a_4ef6_bb19_80762128b240.slice/crio-003fcf82731a42019f54baac7296fd9aeba0f97bbf4b9fe034224025e16ecf84 WatchSource:0}: Error finding container 003fcf82731a42019f54baac7296fd9aeba0f97bbf4b9fe034224025e16ecf84: Status 404 returned error can't find the container with id 003fcf82731a42019f54baac7296fd9aeba0f97bbf4b9fe034224025e16ecf84 Nov 01 20:31:12.467073 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:12.467049 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7367-d6cf8b" event={"ID":"68d66926-3d5a-4ef6-bb19-80762128b240","Type":"ContainerStarted","Data":"60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00"} Nov 01 20:31:12.467179 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:12.467088 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7367-d6cf8b" event={"ID":"68d66926-3d5a-4ef6-bb19-80762128b240","Type":"ContainerStarted","Data":"003fcf82731a42019f54baac7296fd9aeba0f97bbf4b9fe034224025e16ecf84"} Nov 01 20:31:12.576975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:12.576870 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7367-d6cf8b" podStartSLOduration=1.576855388 podStartE2EDuration="1.576855388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:12.576083656 +0000 UTC m=+1673.520921525" watchObservedRunningTime="2026-11-01 20:31:12.576855388 +0000 UTC m=+1673.521693256" Nov 01 20:31:13.967148 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:13.967116 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7367-d6cf8b"] Nov 01 20:31:13.970177 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:13.970145 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7367-d6cf8b"] Nov 01 20:31:14.477342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.477306 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7367-d6cf8b" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" containerName="test-curl-7367-d6cf8b" containerID="cri-o://60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00" gracePeriod=2 Nov 01 20:31:14.528004 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:14.527972 5444 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7367-d6cf8b\" is forbidden: User \"system:node:ip-10-0-138-177.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-177.us-west-1.compute.internal' and this object" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:14.818781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.818753 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn"] Nov 01 20:31:14.819173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.819161 5444 cpu_manager.go:414] "containerMap: removing container" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" containerName="test-curl-7367-d6cf8b" Nov 01 20:31:14.819213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.819174 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" containerName="test-curl-7367-d6cf8b" Nov 01 20:31:14.819275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.819267 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" containerName="test-curl-7367-d6cf8b" Nov 01 20:31:14.822383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.822368 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.873005 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:14.872979 5444 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7367-d6cf8b\" is forbidden: User \"system:node:ip-10-0-138-177.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-177.us-west-1.compute.internal' and this object" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:14.924181 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.924155 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn"] Nov 01 20:31:14.940593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940567 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.940751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940597 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.940751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940626 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.940879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940765 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.940879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940796 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.940879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940864 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.941050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940903 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.941050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940925 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klblb\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:14.941050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:14.940994 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.041559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041533 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.041559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041564 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041585 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041619 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041659 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041703 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041724 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041746 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klblb\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.041794 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.042046 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042388 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.042072 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.042475 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042590 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.042565 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.042957 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.042934 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.044095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.044073 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.044216 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.044201 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.098142 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.098088 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.098414 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.098393 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klblb\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb\") pod \"e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.135219 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.135199 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:15.559087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.559063 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn"] Nov 01 20:31:15.561182 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:31:15.561149 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec99b9b_1bfa_4049_8d7a_b9e22107165e.slice/crio-1eb48c9ac76109e442def300101b473a59faf4654095ac5127ca8da0b736d877 WatchSource:0}: Error finding container 1eb48c9ac76109e442def300101b473a59faf4654095ac5127ca8da0b736d877: Status 404 returned error can't find the container with id 1eb48c9ac76109e442def300101b473a59faf4654095ac5127ca8da0b736d877 Nov 01 20:31:15.563340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.563310 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:15.563395 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.563370 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:15.563442 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:15.563399 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:16.488266 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:16.488226 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" event={"ID":"aec99b9b-1bfa-4049-8d7a-b9e22107165e","Type":"ContainerStarted","Data":"85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b"} Nov 01 20:31:16.488703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:16.488291 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" event={"ID":"aec99b9b-1bfa-4049-8d7a-b9e22107165e","Type":"ContainerStarted","Data":"1eb48c9ac76109e442def300101b473a59faf4654095ac5127ca8da0b736d877"} Nov 01 20:31:16.694434 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:16.694407 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:16.774023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:16.773990 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz\" (UniqueName: \"kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz\") pod \"68d66926-3d5a-4ef6-bb19-80762128b240\" (UID: \"68d66926-3d5a-4ef6-bb19-80762128b240\") " Nov 01 20:31:16.776111 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:16.776079 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz" pod "68d66926-3d5a-4ef6-bb19-80762128b240" (UID: "68d66926-3d5a-4ef6-bb19-80762128b240"). InnerVolumeSpecName "kube-api-access-rdrgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:16.875084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:16.875031 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdrgz\" (UniqueName: \"kubernetes.io/projected/68d66926-3d5a-4ef6-bb19-80762128b240-kube-api-access-rdrgz\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:17.135436 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.135361 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:17.136494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.136472 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:31:17.136560 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.136517 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:31:17.174627 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.174596 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d66926-3d5a-4ef6-bb19-80762128b240" path="/var/lib/kubelet/pods/68d66926-3d5a-4ef6-bb19-80762128b240/volumes" Nov 01 20:31:17.494052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.493973 5444 generic.go:356] "Generic (PLEG): container finished" podID="68d66926-3d5a-4ef6-bb19-80762128b240" containerID="60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00" exitCode=137 Nov 01 20:31:17.494052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.494013 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7367-d6cf8b" Nov 01 20:31:17.494559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.494072 5444 scope.go:122] "RemoveContainer" containerID="60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00" Nov 01 20:31:17.502516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.502499 5444 scope.go:122] "RemoveContainer" containerID="60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00" Nov 01 20:31:17.502813 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:17.502795 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00\": container with ID starting with 60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00 not found: ID does not exist" containerID="60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00" Nov 01 20:31:17.502892 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:17.502820 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00"} err="failed to get container status \"60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00\": rpc error: code = NotFound desc = could not find container \"60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00\": container with ID starting with 60075ed4d6dfc5e64ef3115b6267be4219b28814bf1391219b03f73f7c197e00 not found: ID does not exist" Nov 01 20:31:18.135504 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.135474 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:31:18.135689 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.135521 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:31:18.198784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.198734 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podStartSLOduration=4.198720882 podStartE2EDuration="4.198720882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:16.669880679 +0000 UTC m=+1677.614718546" watchObservedRunningTime="2026-11-01 20:31:18.198720882 +0000 UTC m=+1679.143558757" Nov 01 20:31:18.199234 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.199220 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz"] Nov 01 20:31:18.199416 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.199396 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" podUID="9dfdf434-14d0-4d38-a9c2-97473d349459" containerName="maas-api" containerID="cri-o://9406f160fa934a43b112582e04697d953e02010541e463532415898f8983bf35" gracePeriod=30 Nov 01 20:31:18.778286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.777367 5444 generic.go:356] "Generic (PLEG): container finished" podID="9dfdf434-14d0-4d38-a9c2-97473d349459" containerID="9406f160fa934a43b112582e04697d953e02010541e463532415898f8983bf35" exitCode=0 Nov 01 20:31:18.778286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.777826 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" event={"ID":"9dfdf434-14d0-4d38-a9c2-97473d349459","Type":"ContainerDied","Data":"9406f160fa934a43b112582e04697d953e02010541e463532415898f8983bf35"} Nov 01 20:31:18.782386 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.782181 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:31:18.975511 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.975430 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\") pod \"9dfdf434-14d0-4d38-a9c2-97473d349459\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " Nov 01 20:31:18.975511 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.975484 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn\" (UniqueName: \"kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn\") pod \"9dfdf434-14d0-4d38-a9c2-97473d349459\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " Nov 01 20:31:18.975511 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.975512 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs\") pod \"9dfdf434-14d0-4d38-a9c2-97473d349459\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " Nov 01 20:31:18.975834 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.975554 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca\") pod \"9dfdf434-14d0-4d38-a9c2-97473d349459\" (UID: \"9dfdf434-14d0-4d38-a9c2-97473d349459\") " Nov 01 20:31:18.975998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.975966 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca" pod "9dfdf434-14d0-4d38-a9c2-97473d349459" (UID: "9dfdf434-14d0-4d38-a9c2-97473d349459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:31:18.977734 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.977707 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs" pod "9dfdf434-14d0-4d38-a9c2-97473d349459" (UID: "9dfdf434-14d0-4d38-a9c2-97473d349459"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:18.977843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.977749 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls" pod "9dfdf434-14d0-4d38-a9c2-97473d349459" (UID: "9dfdf434-14d0-4d38-a9c2-97473d349459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:18.977843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:18.977755 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn" pod "9dfdf434-14d0-4d38-a9c2-97473d349459" (UID: "9dfdf434-14d0-4d38-a9c2-97473d349459"). InnerVolumeSpecName "kube-api-access-4gcnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:19.077287 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.077254 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:19.077287 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.077282 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcnn\" (UniqueName: \"kubernetes.io/projected/9dfdf434-14d0-4d38-a9c2-97473d349459-kube-api-access-4gcnn\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:19.077456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.077295 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfdf434-14d0-4d38-a9c2-97473d349459-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:19.077456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.077308 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfdf434-14d0-4d38-a9c2-97473d349459-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:31:19.137661 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.137187 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:31:19.137661 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.137253 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:31:19.784040 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.784005 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" event={"ID":"9dfdf434-14d0-4d38-a9c2-97473d349459","Type":"ContainerDied","Data":"69e10c8a0f400964852238c730af1c7277223ec172fdc293419826a8ed4caa4e"} Nov 01 20:31:19.784481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.784043 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz" Nov 01 20:31:19.784481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.784088 5444 scope.go:122] "RemoveContainer" containerID="9406f160fa934a43b112582e04697d953e02010541e463532415898f8983bf35" Nov 01 20:31:19.970801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.970706 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz"] Nov 01 20:31:19.971858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:19.971836 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-202d23-6d5ccc7ff5-vfmbz"] Nov 01 20:31:20.136155 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:20.136120 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:31:20.136330 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:20.136185 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:31:21.135799 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:21.135766 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:31:21.136179 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:21.135843 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:31:21.171899 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:21.171867 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dfdf434-14d0-4d38-a9c2-97473d349459" path="/var/lib/kubelet/pods/9dfdf434-14d0-4d38-a9c2-97473d349459/volumes" Nov 01 20:31:22.202981 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:22.202943 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:22.203475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:22.203346 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:22.204806 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:22.204785 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:31:24.734133 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.734101 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7"] Nov 01 20:31:24.734575 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.734561 5444 cpu_manager.go:414] "containerMap: removing container" podUID="9dfdf434-14d0-4d38-a9c2-97473d349459" containerName="maas-api" Nov 01 20:31:24.734615 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.734576 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dfdf434-14d0-4d38-a9c2-97473d349459" containerName="maas-api" Nov 01 20:31:24.734685 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.734674 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dfdf434-14d0-4d38-a9c2-97473d349459" containerName="maas-api" Nov 01 20:31:24.751476 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.751441 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.796097 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.796068 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7"] Nov 01 20:31:24.842941 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.842912 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-393dc263-serving-cert\"" Nov 01 20:31:24.847516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.847480 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57rd\" (UniqueName: \"kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.847516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.847516 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.847768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.847694 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.847768 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.847739 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.948468 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.948438 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57rd\" (UniqueName: \"kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.948708 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.948477 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.948708 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.948564 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.948708 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.948675 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.948708 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:24.948696 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-serving-cert: secret "maas-api-e2e-ait-393dc263-serving-cert" not found Nov 01 20:31:24.948965 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:24.948787 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls podName:098c1981-5e10-4452-82da-6b8544adc48f nodeName:}" failed. No retries permitted until 2026-11-01 20:31:25.448762228 +0000 UTC m=+1686.393600093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls") pod "maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" (UID: "098c1981-5e10-4452-82da-6b8544adc48f") : secret "maas-api-e2e-ait-393dc263-serving-cert" not found Nov 01 20:31:24.949436 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.949406 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:24.951780 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:24.951755 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:25.005484 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:25.005417 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57rd\" (UniqueName: \"kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:25.467100 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:25.467067 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:25.469753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:25.469728 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\") pod \"maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:25.680195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:25.680163 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:26.051495 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:26.051313 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7"] Nov 01 20:31:26.054566 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:31:26.054534 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098c1981_5e10_4452_82da_6b8544adc48f.slice/crio-4dd9779b316e8a9d616ff2f08073296b54f146bd8554950ca716de9365636c65 WatchSource:0}: Error finding container 4dd9779b316e8a9d616ff2f08073296b54f146bd8554950ca716de9365636c65: Status 404 returned error can't find the container with id 4dd9779b316e8a9d616ff2f08073296b54f146bd8554950ca716de9365636c65 Nov 01 20:31:26.056015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:26.055993 5444 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:31:26.821539 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:26.821505 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" event={"ID":"098c1981-5e10-4452-82da-6b8544adc48f","Type":"ContainerStarted","Data":"4dd9779b316e8a9d616ff2f08073296b54f146bd8554950ca716de9365636c65"} Nov 01 20:31:27.829285 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:27.829248 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" event={"ID":"098c1981-5e10-4452-82da-6b8544adc48f","Type":"ContainerStarted","Data":"546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964"} Nov 01 20:31:27.829668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:27.829421 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:27.943679 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:27.943614 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" podStartSLOduration=3.063528273 podStartE2EDuration="3.943600416s" totalImagesPullingTime="880.072143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:27.941536103 +0000 UTC m=+1688.886373973" watchObservedRunningTime="2026-11-01 20:31:27.943600416 +0000 UTC m=+1688.888438283" Nov 01 20:31:30.034300 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:31:30.034260 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:31:33.839958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.839928 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:31:33.861271 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.861243 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8"] Nov 01 20:31:33.864984 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.864964 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.910870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.910831 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd"] Nov 01 20:31:33.914365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.914349 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.968285 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.968255 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8"] Nov 01 20:31:33.970712 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970670 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.970849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970744 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cv8\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.970849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970782 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.970849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970811 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970850 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970875 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970927 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970969 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6xf\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.970994 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971023 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971124 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971172 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971195 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971218 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971259 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971289 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:33.971407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971366 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:33.971407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:33.971394 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.018963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.018930 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd"] Nov 01 20:31:34.072222 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072193 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.072396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072234 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.072396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072259 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.072396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072368 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.072582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072414 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.072582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072453 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.072582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072499 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.072582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072527 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.072582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072564 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.072871 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072848 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.072940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072877 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.072972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072930 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cv8\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072982 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.072992 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073006 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073167 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073013 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073167 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073062 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073167 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073094 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073167 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073137 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073168 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6xf\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073202 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073374 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073234 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073770 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073425 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073770 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073529 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073770 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073688 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073919 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073843 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.073919 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073900 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.073986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.073952 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.075045 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.075020 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.075230 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.075210 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.075336 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.075316 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.075419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.075402 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.214379 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.214256 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.215117 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.215067 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.217628 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.217601 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cv8\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.218297 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.218252 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6xf\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf\") pod \"e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.234118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.229938 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:34.477813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.477737 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:34.660147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.660124 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd"] Nov 01 20:31:34.661986 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:31:34.661950 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0efa394_28b5_4bb8_9c4f_ecf50fbe29c9.slice/crio-72d0e6a7c8ef2eea8f03c6e4af87fe1e29f30e79b553031a2bd5f86dd14be1ff WatchSource:0}: Error finding container 72d0e6a7c8ef2eea8f03c6e4af87fe1e29f30e79b553031a2bd5f86dd14be1ff: Status 404 returned error can't find the container with id 72d0e6a7c8ef2eea8f03c6e4af87fe1e29f30e79b553031a2bd5f86dd14be1ff Nov 01 20:31:34.664441 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.664406 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:34.664525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.664471 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:34.664525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.664497 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:34.850721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.850691 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8"] Nov 01 20:31:34.855275 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:31:34.855242 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1596f231_22e8_40ee_ad8f_3368f46b2042.slice/crio-1229c5708ebb4707c6bb3e6f81c1dec5cd40cd5653b1a65ba3d174f06775ff8f WatchSource:0}: Error finding container 1229c5708ebb4707c6bb3e6f81c1dec5cd40cd5653b1a65ba3d174f06775ff8f: Status 404 returned error can't find the container with id 1229c5708ebb4707c6bb3e6f81c1dec5cd40cd5653b1a65ba3d174f06775ff8f Nov 01 20:31:34.858194 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.858158 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:34.858311 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.858226 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:34.858311 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.858256 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:31:34.865867 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.865805 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" event={"ID":"1596f231-22e8-40ee-ad8f-3368f46b2042","Type":"ContainerStarted","Data":"1229c5708ebb4707c6bb3e6f81c1dec5cd40cd5653b1a65ba3d174f06775ff8f"} Nov 01 20:31:34.867885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.867864 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" event={"ID":"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9","Type":"ContainerStarted","Data":"5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0"} Nov 01 20:31:34.868005 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:34.867910 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" event={"ID":"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9","Type":"ContainerStarted","Data":"72d0e6a7c8ef2eea8f03c6e4af87fe1e29f30e79b553031a2bd5f86dd14be1ff"} Nov 01 20:31:35.005408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:35.004305 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podStartSLOduration=2.004289628 podStartE2EDuration="2.004289628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:35.002360915 +0000 UTC m=+1695.947198784" watchObservedRunningTime="2026-11-01 20:31:35.004289628 +0000 UTC m=+1695.949127495" Nov 01 20:31:35.234356 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:35.234275 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:35.235546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:35.235521 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:31:35.235662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:35.235570 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:31:35.904195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:35.904155 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" event={"ID":"1596f231-22e8-40ee-ad8f-3368f46b2042","Type":"ContainerStarted","Data":"335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8"} Nov 01 20:31:36.021850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:36.021779 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podStartSLOduration=3.021760701 podStartE2EDuration="3.021760701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:36.01957023 +0000 UTC m=+1696.964408097" watchObservedRunningTime="2026-11-01 20:31:36.021760701 +0000 UTC m=+1696.966598569" Nov 01 20:31:36.235532 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:36.235448 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:31:36.235719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:36.235522 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:31:36.484661 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:36.483803 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:36.485733 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:36.485661 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:31:36.486476 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:36.486441 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:31:37.234896 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:37.234864 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:31:37.235365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:37.234930 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:31:37.489719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:37.489053 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:31:37.489719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:37.489115 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:31:38.235120 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:38.235087 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:31:38.235509 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:38.235139 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:31:38.483665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:38.478995 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:31:38.483665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:38.479060 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:31:39.234442 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:39.234409 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:31:39.234621 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:39.234463 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:31:39.485725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:39.484947 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:31:39.485725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:39.485008 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:31:40.235447 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:40.235414 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:31:40.235620 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:40.235462 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:31:40.479001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:40.478962 5444 patch_prober.go:29] interesting pod/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:31:40.480118 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:40.480082 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:31:41.315247 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:41.315203 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:41.315750 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:41.315507 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:41.316404 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:41.316384 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:31:41.504784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:41.504747 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:41.505197 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:41.505174 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:31:41.505961 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:31:41.505933 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:32:00.036226 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:32:00.036190 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:32:13.050615 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.050577 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp"] Nov 01 20:32:13.054364 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.054344 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.106004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.105977 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-393dc263-openshift-default-dockercfg-ssfv2\"" Nov 01 20:32:13.122795 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.122763 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.122952 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.122811 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.122952 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.122913 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.123046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.122964 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.123046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.122987 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.123046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.123028 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hjd\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.123162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.123085 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.123162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.123103 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.123162 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.123131 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224264 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224217 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224285 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224316 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224366 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224423 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224448 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224498 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hjd\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224533 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224745 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224558 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.224968 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.224858 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.225107 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.225081 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.225248 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.225193 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.225402 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.225381 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.225566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.225540 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.227056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.227031 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.227056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.227043 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.260033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.260002 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp"] Nov 01 20:32:13.280954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.280922 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.281097 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.281004 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hjd\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-6prvp\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.367456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.367378 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:13.798891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.798866 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp"] Nov 01 20:32:13.801330 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:32:13.801301 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b843e4e_be2d_4508_a254_520e7bce7453.slice/crio-45e24f4ed9f24c0f12d5eac605435d1f0079611726411320bd7604604f089933 WatchSource:0}: Error finding container 45e24f4ed9f24c0f12d5eac605435d1f0079611726411320bd7604604f089933: Status 404 returned error can't find the container with id 45e24f4ed9f24c0f12d5eac605435d1f0079611726411320bd7604604f089933 Nov 01 20:32:13.803763 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.803729 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:13.803857 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.803793 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:13.803857 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:13.803821 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:14.144653 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:14.144565 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" event={"ID":"3b843e4e-be2d-4508-a254-520e7bce7453","Type":"ContainerStarted","Data":"75754b12b4a69906b3d4d6121b04b9196a24ce87785787c6886777868ac39c9a"} Nov 01 20:32:14.144653 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:14.144618 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" event={"ID":"3b843e4e-be2d-4508-a254-520e7bce7453","Type":"ContainerStarted","Data":"45e24f4ed9f24c0f12d5eac605435d1f0079611726411320bd7604604f089933"} Nov 01 20:32:14.261448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:14.261378 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podStartSLOduration=1.261363716 podStartE2EDuration="1.261363716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:14.258475747 +0000 UTC m=+1735.203313629" watchObservedRunningTime="2026-11-01 20:32:14.261363716 +0000 UTC m=+1735.206201584" Nov 01 20:32:14.369820 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:14.369761 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:14.371571 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:14.371547 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:32:14.371652 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:14.371599 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 20:32:15.376613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:15.376570 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:32:15.380067 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:15.377920 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 20:32:16.375751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:16.370037 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:32:16.375751 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:16.375239 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 20:32:16.941863 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:16.941830 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8f4cfbc5-6pmzx"] Nov 01 20:32:16.951662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:16.945680 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.004133 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.004105 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f4cfbc5-6pmzx"] Nov 01 20:32:17.039916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.039891 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040036 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.039921 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040036 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.039943 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040107 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.040066 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040107 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.040092 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040175 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.040121 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040234 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.040222 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncjv\" (UniqueName: \"kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.040344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.040332 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.070002 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.069968 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 20:32:17.070169 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.070081 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 20:32:17.070169 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.070116 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 20:32:17.070169 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.069968 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 20:32:17.070169 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.070148 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 20:32:17.070405 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.070262 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 20:32:17.070405 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.070352 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-877r8\"" Nov 01 20:32:17.144331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.144297 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncjv\" (UniqueName: \"kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.148849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.148820 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.149068 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.149049 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.149184 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.149170 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.167658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.149284 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.167887 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.167864 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.168007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.167992 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.168131 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.168116 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.168664 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.168626 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.168795 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.161257 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.169505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.169483 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.170203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.170180 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.174791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.174766 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 20:32:17.175103 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.175081 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 20:32:17.175382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.175354 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.178430 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.178402 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.181532 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.181507 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.217109 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.217063 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncjv\" (UniqueName: \"kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv\") pod \"console-8f4cfbc5-6pmzx\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.276004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.275977 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:17.372369 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.371918 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:32:17.372369 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.371980 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 20:32:17.665660 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:17.655301 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8f4cfbc5-6pmzx"] Nov 01 20:32:17.685168 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:32:17.685141 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f98f7e_1279_42bc_adb3_352cf12792fd.slice/crio-94594265b51be9f1ebcf596407872989a1fdf1e7ddc9f11905695297e153b155 WatchSource:0}: Error finding container 94594265b51be9f1ebcf596407872989a1fdf1e7ddc9f11905695297e153b155: Status 404 returned error can't find the container with id 94594265b51be9f1ebcf596407872989a1fdf1e7ddc9f11905695297e153b155 Nov 01 20:32:18.197961 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:18.197927 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f4cfbc5-6pmzx" event={"ID":"35f98f7e-1279-42bc-adb3-352cf12792fd","Type":"ContainerStarted","Data":"3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002"} Nov 01 20:32:18.198315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:18.197979 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f4cfbc5-6pmzx" event={"ID":"35f98f7e-1279-42bc-adb3-352cf12792fd","Type":"ContainerStarted","Data":"94594265b51be9f1ebcf596407872989a1fdf1e7ddc9f11905695297e153b155"} Nov 01 20:32:18.339458 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:18.339411 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8f4cfbc5-6pmzx" podStartSLOduration=2.339396932 podStartE2EDuration="2.339396932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:18.33657657 +0000 UTC m=+1739.281414438" watchObservedRunningTime="2026-11-01 20:32:18.339396932 +0000 UTC m=+1739.284234853" Nov 01 20:32:18.369992 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:18.369962 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:32:18.370125 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:18.370009 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 20:32:19.370050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:19.370016 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 01 20:32:19.370433 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:19.370065 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 01 20:32:20.372024 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:20.371996 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:20.372412 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:20.372267 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:20.372927 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:20.372909 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:32:27.276781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:27.276735 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:27.277412 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:27.276826 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:27.285198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:27.285171 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:28.185407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.185371 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s"] Nov 01 20:32:28.190761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.190737 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236475 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236532 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236666 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236695 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236737 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236771 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236801 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrnl\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236883 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.237161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.236948 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.267545 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.267504 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:32:28.302754 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.302722 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s"] Nov 01 20:32:28.338668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.338621 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.338882 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.338679 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrnl\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.338882 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.338753 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.338882 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.338824 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339227 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339426 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339341 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339463 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339594 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339571 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339607 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339731 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339749 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.339900 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.339787 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.340286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.340262 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.340396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.340350 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.341316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.341296 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.342394 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.342373 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.396317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.396289 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.396570 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.396548 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrnl\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-lq25s\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.503548 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.503462 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:28.937370 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.937341 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s"] Nov 01 20:32:28.940456 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:32:28.940426 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1730cd0_0b75_4717_83d4_94c8d26f7186.slice/crio-5e416f2852d9a41ad96cb96ef0b980e4f4c86fd97e639f7794c4652d8c2b89a3 WatchSource:0}: Error finding container 5e416f2852d9a41ad96cb96ef0b980e4f4c86fd97e639f7794c4652d8c2b89a3: Status 404 returned error can't find the container with id 5e416f2852d9a41ad96cb96ef0b980e4f4c86fd97e639f7794c4652d8c2b89a3 Nov 01 20:32:28.942913 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.942877 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:28.942991 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.942942 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:28.942991 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:28.942970 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:29.267712 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:29.267672 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" event={"ID":"a1730cd0-0b75-4717-83d4-94c8d26f7186","Type":"ContainerStarted","Data":"00048c3cadb625e34e73b66e41c9d9c09786f1b2ab0e9c619aad537cc03524a6"} Nov 01 20:32:29.267911 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:29.267746 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" event={"ID":"a1730cd0-0b75-4717-83d4-94c8d26f7186","Type":"ContainerStarted","Data":"5e416f2852d9a41ad96cb96ef0b980e4f4c86fd97e639f7794c4652d8c2b89a3"} Nov 01 20:32:29.389178 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:29.389119 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podStartSLOduration=1.389100362 podStartE2EDuration="1.389100362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:29.387552724 +0000 UTC m=+1750.332390594" watchObservedRunningTime="2026-11-01 20:32:29.389100362 +0000 UTC m=+1750.333938232" Nov 01 20:32:29.503956 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:29.503922 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:30.035746 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:32:30.035706 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:32:30.506032 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:30.505952 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-lq25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:32:30.506032 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:30.506006 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:32:31.504556 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:31.504522 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-lq25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:32:31.504790 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:31.504584 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:32:32.504232 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:32.504189 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-lq25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:32:32.504624 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:32.504255 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:32:33.504529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:33.504494 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-lq25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:32:33.504948 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:33.504544 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:32:34.504030 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:34.503994 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-lq25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 01 20:32:34.504210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:34.504054 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 01 20:32:35.508480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:35.508450 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:35.508947 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:35.508920 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:35.509474 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:35.509455 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:32:40.466147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:40.466107 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f4cfbc5-6pmzx"] Nov 01 20:32:43.288779 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.288748 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2"] Nov 01 20:32:43.292543 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.292525 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.396276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.396252 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2"] Nov 01 20:32:43.429161 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429127 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rk5\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429176 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429357 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429330 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429413 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429397 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429497 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429473 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429525 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429714 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429695 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429770 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429735 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.429833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.429816 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.530761 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530737 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.530879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530770 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.530879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530796 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.530879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530820 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rk5\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.530879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530839 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530885 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530918 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530944 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.530980 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.531179 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.531289 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.531308 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531478 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.531382 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.531721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.531704 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.533262 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.533241 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.533619 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.533596 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.588547 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.588499 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rk5\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.588630 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.588540 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token\") pod \"e2e-ait-393dc263-openshift-default-c6d966759-kncd2\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:43.606905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:43.606887 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:44.034195 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.034165 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2"] Nov 01 20:32:44.036366 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:32:44.036339 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42406d8_244a_4a52_b935_f90a90894ce9.slice/crio-1015b59a832eae25e3448a1b432c576a4b9e3e467a68018ca26bffbae27cbda3 WatchSource:0}: Error finding container 1015b59a832eae25e3448a1b432c576a4b9e3e467a68018ca26bffbae27cbda3: Status 404 returned error can't find the container with id 1015b59a832eae25e3448a1b432c576a4b9e3e467a68018ca26bffbae27cbda3 Nov 01 20:32:44.038548 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.038510 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:44.038610 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.038582 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:44.038662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.038650 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:44.349014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.348936 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" event={"ID":"c42406d8-244a-4a52-b935-f90a90894ce9","Type":"ContainerStarted","Data":"c7cc0da6ad77c63e5a4ed624bdbaef38a2fd4670eeb37accd554da1150660f56"} Nov 01 20:32:44.349355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.349015 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" event={"ID":"c42406d8-244a-4a52-b935-f90a90894ce9","Type":"ContainerStarted","Data":"1015b59a832eae25e3448a1b432c576a4b9e3e467a68018ca26bffbae27cbda3"} Nov 01 20:32:44.466944 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.466881 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podStartSLOduration=1.4668632640000001 podStartE2EDuration="1.466863264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:44.464156561 +0000 UTC m=+1765.408994430" watchObservedRunningTime="2026-11-01 20:32:44.466863264 +0000 UTC m=+1765.411701133" Nov 01 20:32:44.607729 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:44.607663 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:45.609344 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:45.609310 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-kncd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:32:45.609752 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:45.609367 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:32:46.607963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:46.607923 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-kncd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:32:46.608168 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:46.607981 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:32:47.607543 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:47.607505 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-kncd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:32:47.607984 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:47.607557 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:32:48.607246 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:48.607207 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-kncd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:32:48.607454 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:48.607283 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:32:49.609684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:49.609627 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-kncd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Nov 01 20:32:49.610382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:49.610343 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Nov 01 20:32:50.612196 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:50.612158 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:50.612710 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:50.612687 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:32:50.613483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:32:50.613462 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:33:00.037535 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:00.037498 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:33:05.486423 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.486387 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8f4cfbc5-6pmzx" podUID="35f98f7e-1279-42bc-adb3-352cf12792fd" containerName="console" containerID="cri-o://3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002" gracePeriod=15 Nov 01 20:33:05.740000 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.739943 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:33:05.844359 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844332 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844369 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv\" (UniqueName: \"kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844405 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844430 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844520 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844593 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844655 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844699 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle\") pod \"35f98f7e-1279-42bc-adb3-352cf12792fd\" (UID: \"35f98f7e-1279-42bc-adb3-352cf12792fd\") " Nov 01 20:33:05.844924 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844755 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:05.844924 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844897 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:05.845131 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.844993 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:05.845131 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.845066 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:05.845131 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.845131 5444 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-oauth-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.845351 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.845145 5444 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.845351 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.845180 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:05.846815 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.846785 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:05.847008 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.846982 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:05.847086 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.847051 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv" pod "35f98f7e-1279-42bc-adb3-352cf12792fd" (UID: "35f98f7e-1279-42bc-adb3-352cf12792fd"). InnerVolumeSpecName "kube-api-access-9ncjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:05.946381 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.946354 5444 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-serving-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.946381 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.946379 5444 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35f98f7e-1279-42bc-adb3-352cf12792fd-tmp\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.946557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.946389 5444 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-console-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.946557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.946398 5444 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35f98f7e-1279-42bc-adb3-352cf12792fd-console-oauth-config\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.946557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.946407 5444 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35f98f7e-1279-42bc-adb3-352cf12792fd-trusted-ca-bundle\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:05.946557 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:05.946416 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ncjv\" (UniqueName: \"kubernetes.io/projected/35f98f7e-1279-42bc-adb3-352cf12792fd-kube-api-access-9ncjv\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:06.491337 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.491306 5444 generic.go:356] "Generic (PLEG): container finished" podID="35f98f7e-1279-42bc-adb3-352cf12792fd" containerID="3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002" exitCode=0 Nov 01 20:33:06.491743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.491360 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8f4cfbc5-6pmzx" Nov 01 20:33:06.491743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.491400 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f4cfbc5-6pmzx" event={"ID":"35f98f7e-1279-42bc-adb3-352cf12792fd","Type":"ContainerDied","Data":"3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002"} Nov 01 20:33:06.491743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.491451 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8f4cfbc5-6pmzx" event={"ID":"35f98f7e-1279-42bc-adb3-352cf12792fd","Type":"ContainerDied","Data":"94594265b51be9f1ebcf596407872989a1fdf1e7ddc9f11905695297e153b155"} Nov 01 20:33:06.491743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.491472 5444 scope.go:122] "RemoveContainer" containerID="3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002" Nov 01 20:33:06.500802 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.500787 5444 scope.go:122] "RemoveContainer" containerID="3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002" Nov 01 20:33:06.501097 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:06.501076 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002\": container with ID starting with 3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002 not found: ID does not exist" containerID="3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002" Nov 01 20:33:06.501145 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.501105 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002"} err="failed to get container status \"3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002\": rpc error: code = NotFound desc = could not find container \"3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002\": container with ID starting with 3f3d7704aa2781d3c5e4211030826c10bbbe6d22c37ef2a977a883f3a7133002 not found: ID does not exist" Nov 01 20:33:06.652632 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.652607 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8f4cfbc5-6pmzx"] Nov 01 20:33:06.702142 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:06.702120 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8f4cfbc5-6pmzx"] Nov 01 20:33:07.158783 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:07.158754 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f98f7e-1279-42bc-adb3-352cf12792fd" path="/var/lib/kubelet/pods/35f98f7e-1279-42bc-adb3-352cf12792fd/volumes" Nov 01 20:33:30.035043 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:30.035004 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:33:44.704373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:44.704340 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7"] Nov 01 20:33:44.704869 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:44.704526 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" podUID="098c1981-5e10-4452-82da-6b8544adc48f" containerName="maas-api" containerID="cri-o://546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964" gracePeriod=30 Nov 01 20:33:45.096661 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.091489 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:33:45.185976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.185946 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs\") pod \"098c1981-5e10-4452-82da-6b8544adc48f\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " Nov 01 20:33:45.186284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.186267 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd\" (UniqueName: \"kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd\") pod \"098c1981-5e10-4452-82da-6b8544adc48f\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " Nov 01 20:33:45.186813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.186795 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\") pod \"098c1981-5e10-4452-82da-6b8544adc48f\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " Nov 01 20:33:45.187007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.186989 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca\") pod \"098c1981-5e10-4452-82da-6b8544adc48f\" (UID: \"098c1981-5e10-4452-82da-6b8544adc48f\") " Nov 01 20:33:45.187416 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.187391 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca" pod "098c1981-5e10-4452-82da-6b8544adc48f" (UID: "098c1981-5e10-4452-82da-6b8544adc48f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:45.187668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.187652 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098c1981-5e10-4452-82da-6b8544adc48f-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:45.188787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.188746 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd" pod "098c1981-5e10-4452-82da-6b8544adc48f" (UID: "098c1981-5e10-4452-82da-6b8544adc48f"). InnerVolumeSpecName "kube-api-access-p57rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:45.189552 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.189530 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls" pod "098c1981-5e10-4452-82da-6b8544adc48f" (UID: "098c1981-5e10-4452-82da-6b8544adc48f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:45.235698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.215862 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs" pod "098c1981-5e10-4452-82da-6b8544adc48f" (UID: "098c1981-5e10-4452-82da-6b8544adc48f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:45.289290 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.289261 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:45.289290 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.289289 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098c1981-5e10-4452-82da-6b8544adc48f-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:45.289470 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.289301 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57rd\" (UniqueName: \"kubernetes.io/projected/098c1981-5e10-4452-82da-6b8544adc48f-kube-api-access-p57rd\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:45.803948 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.803911 5444 generic.go:356] "Generic (PLEG): container finished" podID="098c1981-5e10-4452-82da-6b8544adc48f" containerID="546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964" exitCode=0 Nov 01 20:33:45.804574 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.804529 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" event={"ID":"098c1981-5e10-4452-82da-6b8544adc48f","Type":"ContainerDied","Data":"546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964"} Nov 01 20:33:45.804809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.804769 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" event={"ID":"098c1981-5e10-4452-82da-6b8544adc48f","Type":"ContainerDied","Data":"4dd9779b316e8a9d616ff2f08073296b54f146bd8554950ca716de9365636c65"} Nov 01 20:33:45.805078 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.804940 5444 scope.go:122] "RemoveContainer" containerID="546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964" Nov 01 20:33:45.805304 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.805017 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7" Nov 01 20:33:45.823801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.823776 5444 scope.go:122] "RemoveContainer" containerID="546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964" Nov 01 20:33:45.824180 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:45.824110 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964\": container with ID starting with 546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964 not found: ID does not exist" containerID="546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964" Nov 01 20:33:45.824245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.824194 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964"} err="failed to get container status \"546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964\": rpc error: code = NotFound desc = could not find container \"546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964\": container with ID starting with 546f980b5fc1a1f970f7eecef551eec33c22f4438b1631bd873ecd4c567b5964 not found: ID does not exist" Nov 01 20:33:45.893719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.893690 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x"] Nov 01 20:33:45.893903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.893883 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" podUID="3a6dc35b-ffc2-4e29-800d-f40a62751713" containerName="maas-api" containerID="cri-o://348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995" gracePeriod=30 Nov 01 20:33:45.970093 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.970060 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7"] Nov 01 20:33:45.973479 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:45.973453 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-393dc263-c6c88dddb-6vzj7"] Nov 01 20:33:46.234878 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.234850 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:33:46.331198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.331110 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs\") pod \"3a6dc35b-ffc2-4e29-800d-f40a62751713\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " Nov 01 20:33:46.331443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.331420 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\") pod \"3a6dc35b-ffc2-4e29-800d-f40a62751713\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " Nov 01 20:33:46.331883 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.331842 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca\") pod \"3a6dc35b-ffc2-4e29-800d-f40a62751713\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " Nov 01 20:33:46.332103 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.332089 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr\" (UniqueName: \"kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr\") pod \"3a6dc35b-ffc2-4e29-800d-f40a62751713\" (UID: \"3a6dc35b-ffc2-4e29-800d-f40a62751713\") " Nov 01 20:33:46.339099 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.339058 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca" pod "3a6dc35b-ffc2-4e29-800d-f40a62751713" (UID: "3a6dc35b-ffc2-4e29-800d-f40a62751713"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:46.340846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.340817 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr" pod "3a6dc35b-ffc2-4e29-800d-f40a62751713" (UID: "3a6dc35b-ffc2-4e29-800d-f40a62751713"). InnerVolumeSpecName "kube-api-access-kvxgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:46.341250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.341213 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs" pod "3a6dc35b-ffc2-4e29-800d-f40a62751713" (UID: "3a6dc35b-ffc2-4e29-800d-f40a62751713"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:46.351378 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.351341 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls" pod "3a6dc35b-ffc2-4e29-800d-f40a62751713" (UID: "3a6dc35b-ffc2-4e29-800d-f40a62751713"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:46.434446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.434413 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:46.434446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.434444 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a6dc35b-ffc2-4e29-800d-f40a62751713-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:46.434629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.434455 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a6dc35b-ffc2-4e29-800d-f40a62751713-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:46.434629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.434465 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxgr\" (UniqueName: \"kubernetes.io/projected/3a6dc35b-ffc2-4e29-800d-f40a62751713-kube-api-access-kvxgr\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:46.753358 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753288 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8"] Nov 01 20:33:46.753758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753744 5444 cpu_manager.go:414] "containerMap: removing container" podUID="35f98f7e-1279-42bc-adb3-352cf12792fd" containerName="console" Nov 01 20:33:46.753824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753758 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f98f7e-1279-42bc-adb3-352cf12792fd" containerName="console" Nov 01 20:33:46.753824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753771 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3a6dc35b-ffc2-4e29-800d-f40a62751713" containerName="maas-api" Nov 01 20:33:46.753824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753775 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6dc35b-ffc2-4e29-800d-f40a62751713" containerName="maas-api" Nov 01 20:33:46.753824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753783 5444 cpu_manager.go:414] "containerMap: removing container" podUID="098c1981-5e10-4452-82da-6b8544adc48f" containerName="maas-api" Nov 01 20:33:46.753824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753787 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098c1981-5e10-4452-82da-6b8544adc48f" containerName="maas-api" Nov 01 20:33:46.754000 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753859 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f98f7e-1279-42bc-adb3-352cf12792fd" containerName="console" Nov 01 20:33:46.754000 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753869 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6dc35b-ffc2-4e29-800d-f40a62751713" containerName="maas-api" Nov 01 20:33:46.754000 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.753882 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="098c1981-5e10-4452-82da-6b8544adc48f" containerName="maas-api" Nov 01 20:33:46.757241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.757224 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.809951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.809927 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a675a-openshift-default-dockercfg-85chn\"" Nov 01 20:33:46.813400 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.813371 5444 generic.go:356] "Generic (PLEG): container finished" podID="3a6dc35b-ffc2-4e29-800d-f40a62751713" containerID="348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995" exitCode=0 Nov 01 20:33:46.813552 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.813483 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" Nov 01 20:33:46.813627 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.813578 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" event={"ID":"3a6dc35b-ffc2-4e29-800d-f40a62751713","Type":"ContainerDied","Data":"348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995"} Nov 01 20:33:46.814165 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.813630 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x" event={"ID":"3a6dc35b-ffc2-4e29-800d-f40a62751713","Type":"ContainerDied","Data":"e03654409835a62892df855c03bc4766b277702e32f37c8813d7823e1ccc6282"} Nov 01 20:33:46.814165 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.814046 5444 scope.go:122] "RemoveContainer" containerID="348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995" Nov 01 20:33:46.834137 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.832508 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8"] Nov 01 20:33:46.844095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.844059 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.844412 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.844385 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.940774 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.940847 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.940936 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.940982 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.941022 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.941060 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:46.952659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:46.941107 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729rj\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.005045 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.005006 5444 scope.go:122] "RemoveContainer" containerID="348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995" Nov 01 20:33:47.005514 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:47.005450 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995\": container with ID starting with 348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995 not found: ID does not exist" containerID="348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995" Nov 01 20:33:47.005514 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.005488 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995"} err="failed to get container status \"348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995\": rpc error: code = NotFound desc = could not find container \"348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995\": container with ID starting with 348821b482e592474cae96b0ed6b0b92a1f9be23ef39da1352e19f232a0cd995 not found: ID does not exist" Nov 01 20:33:47.042586 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042561 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.042721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042598 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.042721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042621 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.042816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042745 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.042816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042802 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729rj\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.042949 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042925 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.042964 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043056 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043069 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043127 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043175 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043362 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043290 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043450 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043432 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.043917 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.043893 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.044894 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.044872 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.045219 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.045203 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.081245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.081223 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x"] Nov 01 20:33:47.083166 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.083146 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71335882-5696fdc5d5-fdq2x"] Nov 01 20:33:47.099371 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.099350 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729rj\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.099508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.099494 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.161564 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.161540 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098c1981-5e10-4452-82da-6b8544adc48f" path="/var/lib/kubelet/pods/098c1981-5e10-4452-82da-6b8544adc48f/volumes" Nov 01 20:33:47.161985 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.161970 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6dc35b-ffc2-4e29-800d-f40a62751713" path="/var/lib/kubelet/pods/3a6dc35b-ffc2-4e29-800d-f40a62751713/volumes" Nov 01 20:33:47.370213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.370192 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:47.713712 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.712757 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s"] Nov 01 20:33:47.713712 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.713095 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" containerID="cri-o://00048c3cadb625e34e73b66e41c9d9c09786f1b2ab0e9c619aad537cc03524a6" gracePeriod=30 Nov 01 20:33:47.776725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.768007 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2"] Nov 01 20:33:47.776725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.768720 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" containerID="cri-o://c7cc0da6ad77c63e5a4ed624bdbaef38a2fd4670eeb37accd554da1150660f56" gracePeriod=30 Nov 01 20:33:47.806526 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.806494 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp"] Nov 01 20:33:47.806833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.806791 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" containerID="cri-o://75754b12b4a69906b3d4d6121b04b9196a24ce87785787c6886777868ac39c9a" gracePeriod=30 Nov 01 20:33:47.887510 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.887395 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8"] Nov 01 20:33:47.890125 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:33:47.890095 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc4a5c9_68f0_4237_999e_8567fd9ed9f6.slice/crio-40022cb07513649ac1d150c199a9976bcb0bed12f1a21c3133ffbbd26da9560f WatchSource:0}: Error finding container 40022cb07513649ac1d150c199a9976bcb0bed12f1a21c3133ffbbd26da9560f: Status 404 returned error can't find the container with id 40022cb07513649ac1d150c199a9976bcb0bed12f1a21c3133ffbbd26da9560f Nov 01 20:33:47.892747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.892713 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:47.892855 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.892787 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:47.892855 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:47.892831 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:48.288595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:48.288563 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd"] Nov 01 20:33:48.288869 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:48.288834 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" containerID="cri-o://5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0" gracePeriod=30 Nov 01 20:33:48.299774 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:48.299713 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql"] Nov 01 20:33:48.300303 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:48.300236 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" containerID="cri-o://2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9" gracePeriod=30 Nov 01 20:33:48.307680 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:48.301446 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8"] Nov 01 20:33:48.307680 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:48.301628 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" containerID="cri-o://335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8" gracePeriod=30 Nov 01 20:33:49.030660 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.029421 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" event={"ID":"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6","Type":"ContainerStarted","Data":"4f076151a68602c3bcfc86faa73e705adfe9e210c0507edc521dda3c8cd7cef3"} Nov 01 20:33:49.030660 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.029533 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" event={"ID":"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6","Type":"ContainerStarted","Data":"40022cb07513649ac1d150c199a9976bcb0bed12f1a21c3133ffbbd26da9560f"} Nov 01 20:33:49.372885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.372793 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:49.373210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.373179 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:33:49.373396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.373371 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:33:49.513688 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.512114 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podStartSLOduration=3.51209392 podStartE2EDuration="3.51209392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:49.165089784 +0000 UTC m=+1830.109927653" watchObservedRunningTime="2026-11-01 20:33:49.51209392 +0000 UTC m=+1830.456931792" Nov 01 20:33:49.515229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.515198 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf"] Nov 01 20:33:49.519885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.519863 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534424 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534397 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534531 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534517 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534578 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534630 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534605 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534720 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534793 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534834 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534882 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534866 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv4zb\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.534990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.534973 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.580800 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.580770 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf"] Nov 01 20:33:49.636020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.635953 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636010 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636028 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636051 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636098 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636146 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636191 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv4zb\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636251 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636296 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636544 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636727 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636750 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.636778 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636756 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.637006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.636830 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.638501 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.638479 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.639051 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.639030 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.692380 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.692358 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.692481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.692441 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv4zb\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb\") pod \"e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:49.836908 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:49.836885 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:50.293499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.293471 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf"] Nov 01 20:33:50.295812 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:33:50.295787 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6deeedf8_ee4b_4483_895d_30a4659da793.slice/crio-bf2f7fd87f9796416f9cb01d64ffb40d4c59e72da5721462a7ee49e159c612b9 WatchSource:0}: Error finding container bf2f7fd87f9796416f9cb01d64ffb40d4c59e72da5721462a7ee49e159c612b9: Status 404 returned error can't find the container with id bf2f7fd87f9796416f9cb01d64ffb40d4c59e72da5721462a7ee49e159c612b9 Nov 01 20:33:50.298262 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.298228 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:50.298345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.298294 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:50.298345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.298322 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:50.372666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.370963 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:33:50.372666 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.371030 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:33:50.376217 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.376192 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-6prvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:50.376340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.376244 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:50.519905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.519597 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-lq25s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:50.519905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.519665 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:50.615153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.615068 5444 patch_prober.go:29] interesting pod/e2e-ait-393dc263-openshift-default-c6d966759-kncd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:50.615153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:50.615124 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:51.041587 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.041546 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" event={"ID":"6deeedf8-ee4b-4483-895d-30a4659da793","Type":"ContainerStarted","Data":"bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb"} Nov 01 20:33:51.041772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.041655 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" event={"ID":"6deeedf8-ee4b-4483-895d-30a4659da793","Type":"ContainerStarted","Data":"bf2f7fd87f9796416f9cb01d64ffb40d4c59e72da5721462a7ee49e159c612b9"} Nov 01 20:33:51.374719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.374597 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:33:51.374719 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.374679 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:33:51.796515 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.796442 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podStartSLOduration=2.796419132 podStartE2EDuration="2.796419132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:51.173827512 +0000 UTC m=+1832.118665381" watchObservedRunningTime="2026-11-01 20:33:51.796419132 +0000 UTC m=+1832.741257001" Nov 01 20:33:51.798081 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.798032 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2"] Nov 01 20:33:51.838071 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.838044 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:51.839117 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.839084 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:33:51.839196 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.839127 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:33:51.894449 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.894413 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp"] Nov 01 20:33:51.896033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:51.895998 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" containerID="cri-o://dade670b1c60fcff8f647d8e6ca74b30ecde08c46d0000eab160e8d968fea6c4" gracePeriod=30 Nov 01 20:33:52.068914 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:52.068816 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" containerID="cri-o://cc45803bf68bbbfc23a63b3b7cc55f42844b788e3d6d463429b4f5627b3d7113" gracePeriod=30 Nov 01 20:33:52.372716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:52.372030 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:33:52.372716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:52.372088 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:33:52.837525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:52.837495 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:33:52.837875 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:52.837546 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:33:53.114294 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.114249 5444 generic.go:356] "Generic (PLEG): container finished" podID="c42406d8-244a-4a52-b935-f90a90894ce9" containerID="c7cc0da6ad77c63e5a4ed624bdbaef38a2fd4670eeb37accd554da1150660f56" exitCode=0 Nov 01 20:33:53.114445 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.114369 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" event={"ID":"c42406d8-244a-4a52-b935-f90a90894ce9","Type":"ContainerDied","Data":"c7cc0da6ad77c63e5a4ed624bdbaef38a2fd4670eeb37accd554da1150660f56"} Nov 01 20:33:53.117811 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.117774 5444 generic.go:356] "Generic (PLEG): container finished" podID="3b843e4e-be2d-4508-a254-520e7bce7453" containerID="75754b12b4a69906b3d4d6121b04b9196a24ce87785787c6886777868ac39c9a" exitCode=0 Nov 01 20:33:53.117906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.117844 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" event={"ID":"3b843e4e-be2d-4508-a254-520e7bce7453","Type":"ContainerDied","Data":"75754b12b4a69906b3d4d6121b04b9196a24ce87785787c6886777868ac39c9a"} Nov 01 20:33:53.120059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.120032 5444 generic.go:356] "Generic (PLEG): container finished" podID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerID="00048c3cadb625e34e73b66e41c9d9c09786f1b2ab0e9c619aad537cc03524a6" exitCode=0 Nov 01 20:33:53.120430 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.120409 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" event={"ID":"a1730cd0-0b75-4717-83d4-94c8d26f7186","Type":"ContainerDied","Data":"00048c3cadb625e34e73b66e41c9d9c09786f1b2ab0e9c619aad537cc03524a6"} Nov 01 20:33:53.234542 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.232621 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:33:53.303897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.303372 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:53.303897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.303431 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:53.508980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.508747 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:53.508980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.508812 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:53.528842 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.512969 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:33:53.528842 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.513030 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.611819 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.611913 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.611956 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.611989 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.612045 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.612077 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.612129 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.612209 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.612280 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert\") pod \"c42406d8-244a-4a52-b935-f90a90894ce9\" (UID: \"c42406d8-244a-4a52-b935-f90a90894ce9\") " Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.613181 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.617668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.614091 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.620132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.618621 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "kube-api-access-76rk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.620132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.618811 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.620132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.618934 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.620132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.619002 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.620132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.619293 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.620132 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.619721 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.622711 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.622681 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token" pod "c42406d8-244a-4a52-b935-f90a90894ce9" (UID: "c42406d8-244a-4a52-b935-f90a90894ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.631144 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.631115 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:33:53.639936 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.639910 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:33:53.714089 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714064 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.714303 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714289 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.714405 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714396 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.714518 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714507 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.714594 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714584 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.714692 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714677 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.714813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714800 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.714934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.714920 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.715057 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715040 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.715178 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715164 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.715294 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715280 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.715416 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715402 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.715525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715510 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.715677 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715661 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.715813 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715798 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.715938 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715925 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.716087 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716072 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token\") pod \"3b843e4e-be2d-4508-a254-520e7bce7453\" (UID: \"3b843e4e-be2d-4508-a254-520e7bce7453\") " Nov 01 20:33:53.716196 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716182 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert\") pod \"a1730cd0-0b75-4717-83d4-94c8d26f7186\" (UID: \"a1730cd0-0b75-4717-83d4-94c8d26f7186\") " Nov 01 20:33:53.716847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716828 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42406d8-244a-4a52-b935-f90a90894ce9-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.716975 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716959 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rk5\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-kube-api-access-76rk5\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717130 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717114 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717226 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717213 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42406d8-244a-4a52-b935-f90a90894ce9-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717306 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717402 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717388 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42406d8-244a-4a52-b935-f90a90894ce9-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717488 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717476 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717579 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717759 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717744 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42406d8-244a-4a52-b935-f90a90894ce9-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.717903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715808 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.715988 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718194 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716161 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716337 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718403 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716487 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718481 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.716905 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718560 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.717349 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.718049 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.718756 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.718343 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.718829 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.718563 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.718920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.718851 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:53.720382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.720328 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.720382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.720351 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.721137 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.721104 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:53.721239 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.721173 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "kube-api-access-w4hjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.722499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.722396 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl" pod "a1730cd0-0b75-4717-83d4-94c8d26f7186" (UID: "a1730cd0-0b75-4717-83d4-94c8d26f7186"). InnerVolumeSpecName "kube-api-access-rvrnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.722499 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.722475 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:53.722599 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.722522 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token" pod "3b843e4e-be2d-4508-a254-520e7bce7453" (UID: "3b843e4e-be2d-4508-a254-520e7bce7453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818663 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818693 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b843e4e-be2d-4508-a254-520e7bce7453-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818707 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818719 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818731 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1730cd0-0b75-4717-83d4-94c8d26f7186-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818743 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818760 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818773 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818787 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b843e4e-be2d-4508-a254-520e7bce7453-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818800 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818812 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrnl\" (UniqueName: \"kubernetes.io/projected/a1730cd0-0b75-4717-83d4-94c8d26f7186-kube-api-access-rvrnl\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818826 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818838 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818850 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818862 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b843e4e-be2d-4508-a254-520e7bce7453-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818875 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818888 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hjd\" (UniqueName: \"kubernetes.io/projected/3b843e4e-be2d-4508-a254-520e7bce7453-kube-api-access-w4hjd\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.818928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.818904 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1730cd0-0b75-4717-83d4-94c8d26f7186-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:53.838383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.838104 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:33:53.838383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.838143 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:33:53.932895 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:53.932610 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:33:54.022763 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022733 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.022920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022769 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.022920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022795 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.022920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022824 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.022920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022842 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.022920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022867 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.022920 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022901 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.023100 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022948 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.023100 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.022979 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token\") pod \"1596f231-22e8-40ee-ad8f-3368f46b2042\" (UID: \"1596f231-22e8-40ee-ad8f-3368f46b2042\") " Nov 01 20:33:54.024363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.024336 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.024630 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.024611 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:54.024999 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.024978 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.025483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.025144 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.025483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.025436 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.026366 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.026233 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:54.026366 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.026340 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.027313 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.027264 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "kube-api-access-nx6xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.028027 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.028003 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy" pod "1596f231-22e8-40ee-ad8f-3368f46b2042" (UID: "1596f231-22e8-40ee-ad8f-3368f46b2042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.108007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.107850 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:33:54.109898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.109846 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:33:54.124412 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124380 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.124541 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124452 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124541 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124526 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124569 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124616 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124667 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.124781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124697 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124731 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.124781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124744 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.124986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124788 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.124986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124825 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124856 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124900 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.124986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124943 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.124986 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.124971 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.125277 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125004 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.125277 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125035 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.125277 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125068 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8\") pod \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\" (UID: \"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9\") " Nov 01 20:33:54.125277 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125097 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.125277 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125156 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket\") pod \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\" (UID: \"2a0a1063-881c-489f-9b0d-b3f3a696ad46\") " Nov 01 20:33:54.125579 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125559 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125673 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125584 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125673 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125600 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125673 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125614 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1596f231-22e8-40ee-ad8f-3368f46b2042-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125673 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125628 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125673 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125657 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx6xf\" (UniqueName: \"kubernetes.io/projected/1596f231-22e8-40ee-ad8f-3368f46b2042-kube-api-access-nx6xf\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125673 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125671 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125687 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125701 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125715 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125731 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1596f231-22e8-40ee-ad8f-3368f46b2042-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125094 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125151 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125281 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125357 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125516 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.125980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.125941 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.128530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.127291 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:54.128530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.127801 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "kube-api-access-c9cv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.128530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.127842 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "kube-api-access-bzs6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.128530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.128026 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.128530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.128070 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.128530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.128493 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.128851 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.128629 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.128851 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.128812 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:54.128851 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.128832 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token" pod "2a0a1063-881c-489f-9b0d-b3f3a696ad46" (UID: "2a0a1063-881c-489f-9b0d-b3f3a696ad46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:54.129046 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.129031 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" Nov 01 20:33:54.129084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.129044 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp" event={"ID":"3b843e4e-be2d-4508-a254-520e7bce7453","Type":"ContainerDied","Data":"45e24f4ed9f24c0f12d5eac605435d1f0079611726411320bd7604604f089933"} Nov 01 20:33:54.129130 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.129099 5444 scope.go:122] "RemoveContainer" containerID="75754b12b4a69906b3d4d6121b04b9196a24ce87785787c6886777868ac39c9a" Nov 01 20:33:54.130349 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.130330 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy" pod "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" (UID: "e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:54.130668 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.130619 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" Nov 01 20:33:54.131395 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.130660 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s" event={"ID":"a1730cd0-0b75-4717-83d4-94c8d26f7186","Type":"ContainerDied","Data":"5e416f2852d9a41ad96cb96ef0b980e4f4c86fd97e639f7794c4652d8c2b89a3"} Nov 01 20:33:54.132147 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.132127 5444 generic.go:356] "Generic (PLEG): container finished" podID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerID="2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9" exitCode=0 Nov 01 20:33:54.132250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.132220 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" event={"ID":"2a0a1063-881c-489f-9b0d-b3f3a696ad46","Type":"ContainerDied","Data":"2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9"} Nov 01 20:33:54.132331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.132251 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" event={"ID":"2a0a1063-881c-489f-9b0d-b3f3a696ad46","Type":"ContainerDied","Data":"228923dbc4bae231b2c52ee668836d74341109e91b50837335dfad2817db98d8"} Nov 01 20:33:54.132331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.132310 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql" Nov 01 20:33:54.136938 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.136914 5444 generic.go:356] "Generic (PLEG): container finished" podID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerID="5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0" exitCode=0 Nov 01 20:33:54.137060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.136942 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" event={"ID":"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9","Type":"ContainerDied","Data":"5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0"} Nov 01 20:33:54.137060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.136961 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" Nov 01 20:33:54.137060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.136991 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd" event={"ID":"e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9","Type":"ContainerDied","Data":"72d0e6a7c8ef2eea8f03c6e4af87fe1e29f30e79b553031a2bd5f86dd14be1ff"} Nov 01 20:33:54.139358 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.138848 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" Nov 01 20:33:54.139358 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.138886 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2" event={"ID":"c42406d8-244a-4a52-b935-f90a90894ce9","Type":"ContainerDied","Data":"1015b59a832eae25e3448a1b432c576a4b9e3e467a68018ca26bffbae27cbda3"} Nov 01 20:33:54.141086 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.140345 5444 scope.go:122] "RemoveContainer" containerID="00048c3cadb625e34e73b66e41c9d9c09786f1b2ab0e9c619aad537cc03524a6" Nov 01 20:33:54.141316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.141164 5444 generic.go:356] "Generic (PLEG): container finished" podID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerID="335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8" exitCode=0 Nov 01 20:33:54.141316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.141198 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" event={"ID":"1596f231-22e8-40ee-ad8f-3368f46b2042","Type":"ContainerDied","Data":"335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8"} Nov 01 20:33:54.141316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.141236 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" event={"ID":"1596f231-22e8-40ee-ad8f-3368f46b2042","Type":"ContainerDied","Data":"1229c5708ebb4707c6bb3e6f81c1dec5cd40cd5653b1a65ba3d174f06775ff8f"} Nov 01 20:33:54.141316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.141305 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8" Nov 01 20:33:54.189897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.182757 5444 scope.go:122] "RemoveContainer" containerID="2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9" Nov 01 20:33:54.197483 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.197465 5444 scope.go:122] "RemoveContainer" containerID="2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9" Nov 01 20:33:54.197859 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:54.197739 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9\": container with ID starting with 2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9 not found: ID does not exist" containerID="2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9" Nov 01 20:33:54.197859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.197777 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9"} err="failed to get container status \"2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9\": rpc error: code = NotFound desc = could not find container \"2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9\": container with ID starting with 2521d732bc88bab4434028d9af7233cbcda8c08bc8d71a3872a979715496cdc9 not found: ID does not exist" Nov 01 20:33:54.197859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.197797 5444 scope.go:122] "RemoveContainer" containerID="5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0" Nov 01 20:33:54.205785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.205765 5444 scope.go:122] "RemoveContainer" containerID="5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0" Nov 01 20:33:54.206080 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:54.206063 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0\": container with ID starting with 5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0 not found: ID does not exist" containerID="5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0" Nov 01 20:33:54.206153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.206084 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0"} err="failed to get container status \"5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0\": rpc error: code = NotFound desc = could not find container \"5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0\": container with ID starting with 5efc83fcd9182bb3fbd40c82ef11940dbd259a3fffce20795401bc408bc604d0 not found: ID does not exist" Nov 01 20:33:54.206153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.206098 5444 scope.go:122] "RemoveContainer" containerID="c7cc0da6ad77c63e5a4ed624bdbaef38a2fd4670eeb37accd554da1150660f56" Nov 01 20:33:54.215998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.215983 5444 scope.go:122] "RemoveContainer" containerID="335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8" Nov 01 20:33:54.225425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.225407 5444 scope.go:122] "RemoveContainer" containerID="335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8" Nov 01 20:33:54.225718 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:33:54.225689 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8\": container with ID starting with 335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8 not found: ID does not exist" containerID="335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8" Nov 01 20:33:54.225816 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.225724 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8"} err="failed to get container status \"335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8\": rpc error: code = NotFound desc = could not find container \"335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8\": container with ID starting with 335c4daa5ecfbd20332a943faecd99730c911395fe5334f9a1b4bbd388a6aed8 not found: ID does not exist" Nov 01 20:33:54.226232 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226218 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226236 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226246 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226254 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226263 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226271 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226280 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cv8\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-kube-api-access-c9cv8\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226289 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226297 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226305 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226313 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226321 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226329 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226337 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226345 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a1063-881c-489f-9b0d-b3f3a696ad46-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.226578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.226353 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzs6k\" (UniqueName: \"kubernetes.io/projected/2a0a1063-881c-489f-9b0d-b3f3a696ad46-kube-api-access-bzs6k\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:54.376003 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.375878 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 01 20:33:54.376175 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.376088 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 01 20:33:54.437602 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.437560 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql"] Nov 01 20:33:54.440520 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.440498 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-4n8ql"] Nov 01 20:33:54.574735 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.574675 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd"] Nov 01 20:33:54.575543 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.575522 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-rbxmd"] Nov 01 20:33:54.682785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.682715 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp"] Nov 01 20:33:54.686210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.686185 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-6prvp"] Nov 01 20:33:54.796350 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.796318 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s"] Nov 01 20:33:54.799367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.799339 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-lq25s"] Nov 01 20:33:54.837536 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.837502 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:33:54.837676 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.837566 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:33:54.912072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.907501 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2"] Nov 01 20:33:54.912524 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:54.912391 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-393dc263-openshift-default-c6d966759-kncd2"] Nov 01 20:33:55.020500 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.020463 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8"] Nov 01 20:33:55.024462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.024435 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71335882-openshift-default-f77dbd9c-xkjj8"] Nov 01 20:33:55.177990 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.177956 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" path="/var/lib/kubelet/pods/1596f231-22e8-40ee-ad8f-3368f46b2042/volumes" Nov 01 20:33:55.178933 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.178903 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" path="/var/lib/kubelet/pods/2a0a1063-881c-489f-9b0d-b3f3a696ad46/volumes" Nov 01 20:33:55.179648 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.179615 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" path="/var/lib/kubelet/pods/3b843e4e-be2d-4508-a254-520e7bce7453/volumes" Nov 01 20:33:55.180409 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.180393 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" path="/var/lib/kubelet/pods/a1730cd0-0b75-4717-83d4-94c8d26f7186/volumes" Nov 01 20:33:55.180863 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.180849 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" path="/var/lib/kubelet/pods/c42406d8-244a-4a52-b935-f90a90894ce9/volumes" Nov 01 20:33:55.181261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.181248 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" path="/var/lib/kubelet/pods/e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9/volumes" Nov 01 20:33:55.374674 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.374565 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:55.374933 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.374915 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:55.375670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.375627 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:33:55.837678 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.837622 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:33:55.837850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:55.837708 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:33:56.840036 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:56.840002 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 20:33:56.840530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:56.840070 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 20:33:57.194175 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.194139 5444 generic.go:356] "Generic (PLEG): container finished" podID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerID="dade670b1c60fcff8f647d8e6ca74b30ecde08c46d0000eab160e8d968fea6c4" exitCode=0 Nov 01 20:33:57.194315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.194261 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" event={"ID":"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb","Type":"ContainerDied","Data":"dade670b1c60fcff8f647d8e6ca74b30ecde08c46d0000eab160e8d968fea6c4"} Nov 01 20:33:57.194386 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.194313 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" event={"ID":"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb","Type":"ContainerDied","Data":"186d24d108e0a3ae8931453e4c97e2543f76e2e82bfaf9e2296e5c28188e07c7"} Nov 01 20:33:57.194386 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.194338 5444 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186d24d108e0a3ae8931453e4c97e2543f76e2e82bfaf9e2296e5c28188e07c7" Nov 01 20:33:57.196238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.196216 5444 generic.go:356] "Generic (PLEG): container finished" podID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerID="cc45803bf68bbbfc23a63b3b7cc55f42844b788e3d6d463429b4f5627b3d7113" exitCode=0 Nov 01 20:33:57.197005 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.196983 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" event={"ID":"62999052-8d11-4d48-b297-209dc4e6b4fe","Type":"ContainerDied","Data":"cc45803bf68bbbfc23a63b3b7cc55f42844b788e3d6d463429b4f5627b3d7113"} Nov 01 20:33:57.204431 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.204409 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:33:57.322716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.322693 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:33:57.346905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.346880 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.346940 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.346969 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347033 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347009 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347043 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347075 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347117 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347208 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347150 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347266 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token\") pod \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\" (UID: \"f2d7659e-e0f4-4faa-937f-4fcc487c8dcb\") " Nov 01 20:33:57.347398 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347301 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:57.347508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347413 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.347508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347422 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.347613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347549 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.347613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347594 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.348052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347912 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.348052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347937 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.348052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347952 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.348052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347966 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.348052 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.347981 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.349365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.349333 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:57.349448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.349377 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:57.349625 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.349604 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "kube-api-access-hsm2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:57.349701 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.349615 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy" pod "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" (UID: "f2d7659e-e0f4-4faa-937f-4fcc487c8dcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.448561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448502 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448554 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448762 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448617 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448762 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448677 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448762 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448728 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448832 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448864 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448923 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.448994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.448969 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data\") pod \"62999052-8d11-4d48-b297-209dc4e6b4fe\" (UID: \"62999052-8d11-4d48-b297-209dc4e6b4fe\") " Nov 01 20:33:57.449209 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449080 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.449316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449285 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.449316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449299 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.449511 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449490 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.449588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449508 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:57.449588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449517 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.449588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449540 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.449588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449555 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsm2b\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-kube-api-access-hsm2b\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.449588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449569 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.449588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449584 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.449935 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449598 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.449935 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.449613 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.450823 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.450803 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:57.450944 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.450931 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:57.451301 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.451289 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:57.451434 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.451413 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t" pod "62999052-8d11-4d48-b297-209dc4e6b4fe" (UID: "62999052-8d11-4d48-b297-209dc4e6b4fe"). InnerVolumeSpecName "kube-api-access-5vr2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:57.551135 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.551105 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vr2t\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-kube-api-access-5vr2t\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.551135 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.551131 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.551135 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.551140 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.551345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.551149 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62999052-8d11-4d48-b297-209dc4e6b4fe-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.551345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.551158 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62999052-8d11-4d48-b297-209dc4e6b4fe-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.551345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.551166 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62999052-8d11-4d48-b297-209dc4e6b4fe-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.841822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.841791 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:57.842214 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.842082 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:57.842614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:57.842598 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:33:58.203679 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.203585 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" Nov 01 20:33:58.203679 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.203611 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2" event={"ID":"62999052-8d11-4d48-b297-209dc4e6b4fe","Type":"ContainerDied","Data":"fecf5d49bb42445538a9c7c5269770fc804a3b9720802ee3f141fe6d089116e9"} Nov 01 20:33:58.203891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.203694 5444 scope.go:122] "RemoveContainer" containerID="cc45803bf68bbbfc23a63b3b7cc55f42844b788e3d6d463429b4f5627b3d7113" Nov 01 20:33:58.203891 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.203845 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp" Nov 01 20:33:58.364587 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.364563 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2"] Nov 01 20:33:58.368848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.368827 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-z8gp2"] Nov 01 20:33:58.476213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.476164 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp"] Nov 01 20:33:58.482618 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:58.482598 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-bwqsp"] Nov 01 20:33:59.157519 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:59.157487 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" path="/var/lib/kubelet/pods/62999052-8d11-4d48-b297-209dc4e6b4fe/volumes" Nov 01 20:33:59.157963 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:33:59.157949 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" path="/var/lib/kubelet/pods/f2d7659e-e0f4-4faa-937f-4fcc487c8dcb/volumes" Nov 01 20:34:00.110370 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:34:00.110330 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:34:19.623508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:19.623477 5444 scope.go:122] "RemoveContainer" containerID="dade670b1c60fcff8f647d8e6ca74b30ecde08c46d0000eab160e8d968fea6c4" Nov 01 20:34:24.348302 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:24.348258 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p"] Nov 01 20:34:30.034944 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:34:30.034906 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:34:32.273501 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273468 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv"] Nov 01 20:34:32.273914 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273900 5444 cpu_manager.go:414] "containerMap: removing container" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" Nov 01 20:34:32.273958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273915 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" Nov 01 20:34:32.273958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273928 5444 cpu_manager.go:414] "containerMap: removing container" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" Nov 01 20:34:32.273958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273933 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" Nov 01 20:34:32.273958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273949 5444 cpu_manager.go:414] "containerMap: removing container" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" Nov 01 20:34:32.273958 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273954 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273962 5444 cpu_manager.go:414] "containerMap: removing container" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273967 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273974 5444 cpu_manager.go:414] "containerMap: removing container" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273979 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.273997 5444 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274002 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274008 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274012 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274018 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274022 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274086 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a1063-881c-489f-9b0d-b3f3a696ad46" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274093 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b843e4e-be2d-4508-a254-520e7bce7453" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274099 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1730cd0-0b75-4717-83d4-94c8d26f7186" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274105 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d7659e-e0f4-4faa-937f-4fcc487c8dcb" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274111 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42406d8-244a-4a52-b935-f90a90894ce9" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274118 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="62999052-8d11-4d48-b297-209dc4e6b4fe" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274123 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0efa394-28b5-4bb8-9c4f-ecf50fbe29c9" containerName="istio-proxy" Nov 01 20:34:32.274338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.274128 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="1596f231-22e8-40ee-ad8f-3368f46b2042" containerName="istio-proxy" Nov 01 20:34:32.277247 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.277229 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.328972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.328944 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2f37811a-serving-cert\"" Nov 01 20:34:32.380438 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.380413 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv"] Nov 01 20:34:32.390786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.390758 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.390940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.390856 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.390940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.390917 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.390940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.390934 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjr7\" (UniqueName: \"kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.491631 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.491596 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.491810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.491674 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.491810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.491739 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.491810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.491764 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjr7\" (UniqueName: \"kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.491970 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:34:32.491909 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-serving-cert: secret "maas-api-e2e-derive-2f37811a-serving-cert" not found Nov 01 20:34:32.492029 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:34:32.491995 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls podName:d5b9beb8-7101-4aac-b333-42c2df1996ec nodeName:}" failed. No retries permitted until 2026-11-01 20:34:32.991972582 +0000 UTC m=+1873.936810442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls") pod "maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" (UID: "d5b9beb8-7101-4aac-b333-42c2df1996ec") : secret "maas-api-e2e-derive-2f37811a-serving-cert" not found Nov 01 20:34:32.492272 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.492250 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.494095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.494071 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.570446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.570366 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjr7\" (UniqueName: \"kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:32.997789 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:32.997716 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:33.000064 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:33.000043 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\") pod \"maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:33.189368 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:33.189335 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:33.563771 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:33.563682 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv"] Nov 01 20:34:33.566589 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:34:33.566562 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b9beb8_7101_4aac_b333_42c2df1996ec.slice/crio-4420dc5a8d599e2dc1d38f21f141a1b1b2b981d694d5b31a5dde2ef222e4d955 WatchSource:0}: Error finding container 4420dc5a8d599e2dc1d38f21f141a1b1b2b981d694d5b31a5dde2ef222e4d955: Status 404 returned error can't find the container with id 4420dc5a8d599e2dc1d38f21f141a1b1b2b981d694d5b31a5dde2ef222e4d955 Nov 01 20:34:34.407860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:34.407827 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" event={"ID":"d5b9beb8-7101-4aac-b333-42c2df1996ec","Type":"ContainerStarted","Data":"4420dc5a8d599e2dc1d38f21f141a1b1b2b981d694d5b31a5dde2ef222e4d955"} Nov 01 20:34:35.413830 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:35.413794 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" event={"ID":"d5b9beb8-7101-4aac-b333-42c2df1996ec","Type":"ContainerStarted","Data":"d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221"} Nov 01 20:34:35.414245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:35.413971 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:35.527353 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:35.527297 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" podStartSLOduration=2.631060768 podStartE2EDuration="3.527281597s" totalImagesPullingTime="896.220829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:35.526151341 +0000 UTC m=+1876.470989208" watchObservedRunningTime="2026-11-01 20:34:35.527281597 +0000 UTC m=+1876.472119465" Nov 01 20:34:39.368237 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.368197 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" podUID="929d1b1a-bb8d-470a-abae-a463d511a824" containerName="main" containerID="cri-o://974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075" gracePeriod=45 Nov 01 20:34:39.375131 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.375101 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" podUID="929d1b1a-bb8d-470a-abae-a463d511a824" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused" Nov 01 20:34:39.614089 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.614063 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:34:39.667733 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.667674 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs\" (UniqueName: \"kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs\") pod \"929d1b1a-bb8d-470a-abae-a463d511a824\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " Nov 01 20:34:39.667848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.667754 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir\") pod \"929d1b1a-bb8d-470a-abae-a463d511a824\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " Nov 01 20:34:39.667848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.667785 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm\") pod \"929d1b1a-bb8d-470a-abae-a463d511a824\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " Nov 01 20:34:39.667848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.667806 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h\" (UniqueName: \"kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h\") pod \"929d1b1a-bb8d-470a-abae-a463d511a824\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " Nov 01 20:34:39.667848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.667842 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home\") pod \"929d1b1a-bb8d-470a-abae-a463d511a824\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " Nov 01 20:34:39.668022 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.667914 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache\") pod \"929d1b1a-bb8d-470a-abae-a463d511a824\" (UID: \"929d1b1a-bb8d-470a-abae-a463d511a824\") " Nov 01 20:34:39.668086 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.668054 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir" pod "929d1b1a-bb8d-470a-abae-a463d511a824" (UID: "929d1b1a-bb8d-470a-abae-a463d511a824"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:39.668166 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.668144 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home" pod "929d1b1a-bb8d-470a-abae-a463d511a824" (UID: "929d1b1a-bb8d-470a-abae-a463d511a824"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:39.668257 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.668218 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache" pod "929d1b1a-bb8d-470a-abae-a463d511a824" (UID: "929d1b1a-bb8d-470a-abae-a463d511a824"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:39.668408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.668385 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:39.668508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.668413 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:39.668508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.668425 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:39.669933 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.669914 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h" pod "929d1b1a-bb8d-470a-abae-a463d511a824" (UID: "929d1b1a-bb8d-470a-abae-a463d511a824"). InnerVolumeSpecName "kube-api-access-pnc9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:39.670272 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.670253 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs" pod "929d1b1a-bb8d-470a-abae-a463d511a824" (UID: "929d1b1a-bb8d-470a-abae-a463d511a824"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:34:39.670320 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.670268 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm" pod "929d1b1a-bb8d-470a-abae-a463d511a824" (UID: "929d1b1a-bb8d-470a-abae-a463d511a824"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:34:39.768907 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.768884 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929d1b1a-bb8d-470a-abae-a463d511a824-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:39.768907 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.768906 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929d1b1a-bb8d-470a-abae-a463d511a824-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:39.769044 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:39.768916 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnc9h\" (UniqueName: \"kubernetes.io/projected/929d1b1a-bb8d-470a-abae-a463d511a824-kube-api-access-pnc9h\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:34:40.438543 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.438507 5444 generic.go:356] "Generic (PLEG): container finished" podID="929d1b1a-bb8d-470a-abae-a463d511a824" containerID="974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075" exitCode=0 Nov 01 20:34:40.439079 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.438562 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" Nov 01 20:34:40.439079 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.438592 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" event={"ID":"929d1b1a-bb8d-470a-abae-a463d511a824","Type":"ContainerDied","Data":"974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075"} Nov 01 20:34:40.439079 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.438668 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p" event={"ID":"929d1b1a-bb8d-470a-abae-a463d511a824","Type":"ContainerDied","Data":"a29708d2cdcc8d7431e2aeaf90205eb9f4e9bf00d769b288f27112ffff101bec"} Nov 01 20:34:40.439079 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.438694 5444 scope.go:122] "RemoveContainer" containerID="974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075" Nov 01 20:34:40.447604 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.447584 5444 scope.go:122] "RemoveContainer" containerID="974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075" Nov 01 20:34:40.447882 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:34:40.447861 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075\": container with ID starting with 974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075 not found: ID does not exist" containerID="974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075" Nov 01 20:34:40.447938 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.447890 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075"} err="failed to get container status \"974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075\": rpc error: code = NotFound desc = could not find container \"974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075\": container with ID starting with 974180b0c5b14a170f8a46b9c6e7b60a4ea8f80baeb3036587e291df5c369075 not found: ID does not exist" Nov 01 20:34:40.600873 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.600845 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p"] Nov 01 20:34:40.650215 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:40.650189 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2cc24d/sub-test-model-2cc24d-kserve-66c7c5f5d9-x866p"] Nov 01 20:34:41.157833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:41.157803 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929d1b1a-bb8d-470a-abae-a463d511a824" path="/var/lib/kubelet/pods/929d1b1a-bb8d-470a-abae-a463d511a824/volumes" Nov 01 20:34:41.424833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:41.424764 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:34:47.217009 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.216976 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr"] Nov 01 20:34:47.217603 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.217582 5444 cpu_manager.go:414] "containerMap: removing container" podUID="929d1b1a-bb8d-470a-abae-a463d511a824" containerName="main" Nov 01 20:34:47.217718 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.217605 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929d1b1a-bb8d-470a-abae-a463d511a824" containerName="main" Nov 01 20:34:47.217781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.217762 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="929d1b1a-bb8d-470a-abae-a463d511a824" containerName="main" Nov 01 20:34:47.222077 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.222057 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.334628 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.334597 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr"] Nov 01 20:34:47.383600 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383571 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383646 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383674 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383874 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383773 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383874 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383853 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383978 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383872 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383978 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383894 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.383978 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.383924 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk95\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.384096 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.384029 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487320 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487377 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487414 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487461 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk95\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487521 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487568 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487680 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487714 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.487742 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.488889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.488355 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.489923 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.489889 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.490286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.490267 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.501684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.493244 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.501684 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.494189 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.502226 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.502155 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.527508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.527480 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.562694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.557122 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.562890 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.557794 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk95\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4ghvr\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:47.834540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:47.834498 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:48.384406 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.384372 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr"] Nov 01 20:34:48.385003 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:34:48.384980 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abf0006_4b7d_4789_9417_ae3b2bec1dc5.slice/crio-346c327f77a1b7a09652a87feb7a22db7cddd7ada32aa2bb9b6aa3f8cf57a586 WatchSource:0}: Error finding container 346c327f77a1b7a09652a87feb7a22db7cddd7ada32aa2bb9b6aa3f8cf57a586: Status 404 returned error can't find the container with id 346c327f77a1b7a09652a87feb7a22db7cddd7ada32aa2bb9b6aa3f8cf57a586 Nov 01 20:34:48.387456 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.387416 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:34:48.387552 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.387495 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:34:48.387552 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.387538 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:34:48.527833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.527786 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" event={"ID":"4abf0006-4b7d-4789-9417-ae3b2bec1dc5","Type":"ContainerStarted","Data":"9a2c33e8f1f57ea465d2600c5b175575e0fbe405ba0a16e009458e13fde18c7e"} Nov 01 20:34:48.527992 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.527869 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" event={"ID":"4abf0006-4b7d-4789-9417-ae3b2bec1dc5","Type":"ContainerStarted","Data":"346c327f77a1b7a09652a87feb7a22db7cddd7ada32aa2bb9b6aa3f8cf57a586"} Nov 01 20:34:48.835382 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:48.835339 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:49.836278 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:49.836207 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:49.836278 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:49.836258 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:50.835332 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:50.835296 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:50.835591 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:50.835380 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:51.582774 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:51.582734 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:34:51.583298 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:51.582798 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:34:51.835377 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:51.835301 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:51.835377 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:51.835350 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:52.835649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:52.835609 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:52.836037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:52.835683 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:53.834999 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:53.834960 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:53.835173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:53.835036 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:54.834971 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:54.834936 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:54.835451 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:54.835003 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:55.835067 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:55.835030 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 01 20:34:55.835461 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:55.835093 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 01 20:34:57.846807 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:57.846763 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:34:57.847320 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:57.846822 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:34:57.873068 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:57.873038 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:57.873709 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:57.873682 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:57.874551 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:57.874526 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:34:58.012441 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:34:58.012386 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podStartSLOduration=11.012374269 podStartE2EDuration="11.012374269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:48.653973046 +0000 UTC m=+1889.598810914" watchObservedRunningTime="2026-11-01 20:34:58.012374269 +0000 UTC m=+1898.957212176" Nov 01 20:35:00.037793 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:35:00.037752 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:35:17.511446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.511418 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl"] Nov 01 20:35:17.515014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.514998 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.621871 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.621833 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl"] Nov 01 20:35:17.649583 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649552 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649589 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649613 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfzl\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649738 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649810 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649828 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649850 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649865 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.649940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.649880 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751325 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751301 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751333 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751353 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfzl\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751369 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751416 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751466 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751520 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751546 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751683 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751753 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751810 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.751898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.751889 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.752106 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.752087 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.752201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.752181 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.753858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.753834 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.754399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.754377 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.807781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.807718 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.807897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.807871 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfzl\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-r2grl\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:17.828229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:17.828203 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:18.252071 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.252048 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl"] Nov 01 20:35:18.254561 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:35:18.254530 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d765043_d698_444c_8dc6_1d891c364570.slice/crio-0927f89873c06983679cfec1f022f3df97785173d237022f861e5b7020caebb3 WatchSource:0}: Error finding container 0927f89873c06983679cfec1f022f3df97785173d237022f861e5b7020caebb3: Status 404 returned error can't find the container with id 0927f89873c06983679cfec1f022f3df97785173d237022f861e5b7020caebb3 Nov 01 20:35:18.256860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.256827 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:35:18.256947 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.256890 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:35:18.256947 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.256920 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:35:18.708404 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.708367 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" event={"ID":"8d765043-d698-444c-8dc6-1d891c364570","Type":"ContainerStarted","Data":"32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7"} Nov 01 20:35:18.708801 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.708423 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" event={"ID":"8d765043-d698-444c-8dc6-1d891c364570","Type":"ContainerStarted","Data":"0927f89873c06983679cfec1f022f3df97785173d237022f861e5b7020caebb3"} Nov 01 20:35:18.826247 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.826193 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podStartSLOduration=1.8261775070000001 podStartE2EDuration="1.826177507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:18.824226963 +0000 UTC m=+1919.769064832" watchObservedRunningTime="2026-11-01 20:35:18.826177507 +0000 UTC m=+1919.771015375" Nov 01 20:35:18.828546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:18.828515 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:19.830588 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:19.830558 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:19.830966 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:19.830610 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:20.828626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:20.828597 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:20.828828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:20.828675 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:21.828758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:21.828721 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:21.829144 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:21.828787 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:22.828928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:22.828893 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:22.829411 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:22.828952 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:23.829077 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:23.829039 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:23.829546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:23.829118 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:24.828912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:24.828876 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:24.829117 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:24.828950 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:25.829218 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:25.829183 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 01 20:35:25.829592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:25.829234 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 01 20:35:27.776239 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:27.776210 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:27.776713 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:27.776546 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:27.778062 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:27.778036 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:35:30.034315 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:35:30.034277 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:35:58.976624 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:58.976592 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv"] Nov 01 20:35:58.977201 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:58.976800 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" podUID="d5b9beb8-7101-4aac-b333-42c2df1996ec" containerName="maas-api" containerID="cri-o://d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221" gracePeriod=30 Nov 01 20:35:59.320495 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.320209 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:35:59.460353 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.460312 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs\") pod \"d5b9beb8-7101-4aac-b333-42c2df1996ec\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " Nov 01 20:35:59.460566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.460466 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7\" (UniqueName: \"kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7\") pod \"d5b9beb8-7101-4aac-b333-42c2df1996ec\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " Nov 01 20:35:59.460566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.460539 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca\") pod \"d5b9beb8-7101-4aac-b333-42c2df1996ec\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " Nov 01 20:35:59.460705 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.460591 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\") pod \"d5b9beb8-7101-4aac-b333-42c2df1996ec\" (UID: \"d5b9beb8-7101-4aac-b333-42c2df1996ec\") " Nov 01 20:35:59.460919 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.460895 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca" pod "d5b9beb8-7101-4aac-b333-42c2df1996ec" (UID: "d5b9beb8-7101-4aac-b333-42c2df1996ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:35:59.461094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.461046 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b9beb8-7101-4aac-b333-42c2df1996ec-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.463409 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.463375 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs" pod "d5b9beb8-7101-4aac-b333-42c2df1996ec" (UID: "d5b9beb8-7101-4aac-b333-42c2df1996ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:35:59.463520 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.463398 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls" pod "d5b9beb8-7101-4aac-b333-42c2df1996ec" (UID: "d5b9beb8-7101-4aac-b333-42c2df1996ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:35:59.463928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.463906 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7" pod "d5b9beb8-7101-4aac-b333-42c2df1996ec" (UID: "d5b9beb8-7101-4aac-b333-42c2df1996ec"). InnerVolumeSpecName "kube-api-access-4jjr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:35:59.562305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.562235 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjr7\" (UniqueName: \"kubernetes.io/projected/d5b9beb8-7101-4aac-b333-42c2df1996ec-kube-api-access-4jjr7\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.562305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.562262 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.562305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.562271 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b9beb8-7101-4aac-b333-42c2df1996ec-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:35:59.987739 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.987617 5444 generic.go:356] "Generic (PLEG): container finished" podID="d5b9beb8-7101-4aac-b333-42c2df1996ec" containerID="d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221" exitCode=0 Nov 01 20:35:59.989295 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.989248 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" event={"ID":"d5b9beb8-7101-4aac-b333-42c2df1996ec","Type":"ContainerDied","Data":"d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221"} Nov 01 20:35:59.989443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.989328 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" event={"ID":"d5b9beb8-7101-4aac-b333-42c2df1996ec","Type":"ContainerDied","Data":"4420dc5a8d599e2dc1d38f21f141a1b1b2b981d694d5b31a5dde2ef222e4d955"} Nov 01 20:35:59.989443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.989359 5444 scope.go:122] "RemoveContainer" containerID="d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221" Nov 01 20:35:59.989559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:35:59.989521 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv" Nov 01 20:36:00.008088 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:00.008062 5444 scope.go:122] "RemoveContainer" containerID="d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221" Nov 01 20:36:00.008733 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:36:00.008693 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221\": container with ID starting with d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221 not found: ID does not exist" containerID="d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221" Nov 01 20:36:00.008850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:00.008746 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221"} err="failed to get container status \"d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221\": rpc error: code = NotFound desc = could not find container \"d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221\": container with ID starting with d59bc9ae2dd9c12edbe7a889fe954a1cded6990299056c0bede8bb276a56d221 not found: ID does not exist" Nov 01 20:36:00.037380 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:36:00.037351 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:36:00.153931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:00.153905 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv"] Nov 01 20:36:00.202747 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:00.202725 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f37811a-5cb5c9b6fc-l2pzv"] Nov 01 20:36:01.158216 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:01.158183 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b9beb8-7101-4aac-b333-42c2df1996ec" path="/var/lib/kubelet/pods/d5b9beb8-7101-4aac-b333-42c2df1996ec/volumes" Nov 01 20:36:22.198279 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.198241 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8"] Nov 01 20:36:22.198902 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.198881 5444 cpu_manager.go:414] "containerMap: removing container" podUID="d5b9beb8-7101-4aac-b333-42c2df1996ec" containerName="maas-api" Nov 01 20:36:22.198989 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.198903 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b9beb8-7101-4aac-b333-42c2df1996ec" containerName="maas-api" Nov 01 20:36:22.199076 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.199031 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b9beb8-7101-4aac-b333-42c2df1996ec" containerName="maas-api" Nov 01 20:36:22.203203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.203182 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.232536 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.232515 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.232659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.232545 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.232659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.232578 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.232774 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.232749 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.232822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.232790 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.232822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.232810 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzz79\" (UniqueName: \"kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.265714 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.265672 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-912d17\"/\"openshift-service-ca.crt\"" Nov 01 20:36:22.265860 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.265843 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-912d17\"/\"kube-root-ca.crt\"" Nov 01 20:36:22.315702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.315682 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-912d17\"/\"default-dockercfg-v7qfw\"" Nov 01 20:36:22.317462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.317442 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8"] Nov 01 20:36:22.318309 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.318295 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-912d17\"/\"noten-model-912d17-kserve-self-signed-certs\"" Nov 01 20:36:22.333782 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.333761 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.333846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.333788 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.333846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.333824 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.333923 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.333893 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.333965 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.333930 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.333965 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.333955 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzz79\" (UniqueName: \"kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.334199 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.334179 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.334276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.334221 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.334276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.334265 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.336006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.335990 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.336305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.336288 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.426365 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.426324 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzz79\" (UniqueName: \"kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79\") pod \"noten-model-912d17-kserve-76bddc988f-2mwl8\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.514928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.514896 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:22.889503 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.889479 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8"] Nov 01 20:36:22.892069 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:36:22.892039 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5da6988b_be1e_4766_b8d8_f5aea4a72a5d.slice/crio-a99604fa0d41fad0a654949fabc1b1330c57a844dd3e45d6666dc1fbc63e2997 WatchSource:0}: Error finding container a99604fa0d41fad0a654949fabc1b1330c57a844dd3e45d6666dc1fbc63e2997: Status 404 returned error can't find the container with id a99604fa0d41fad0a654949fabc1b1330c57a844dd3e45d6666dc1fbc63e2997 Nov 01 20:36:22.913660 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.913620 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx"] Nov 01 20:36:22.913849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:22.913813 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" containerID="cri-o://5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9" gracePeriod=30 Nov 01 20:36:23.255861 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:23.255781 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" event={"ID":"5da6988b-be1e-4766-b8d8-f5aea4a72a5d","Type":"ContainerStarted","Data":"7201c7047510cd74cc46ed373d9295e104736308a2caaf34bc5ecc1435e3e4ca"} Nov 01 20:36:23.255861 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:23.255819 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" event={"ID":"5da6988b-be1e-4766-b8d8-f5aea4a72a5d","Type":"ContainerStarted","Data":"a99604fa0d41fad0a654949fabc1b1330c57a844dd3e45d6666dc1fbc63e2997"} Nov 01 20:36:23.368928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:23.368877 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" podStartSLOduration=1.3688625220000001 podStartE2EDuration="1.368862522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:23.367961475 +0000 UTC m=+1984.312799344" watchObservedRunningTime="2026-11-01 20:36:23.368862522 +0000 UTC m=+1984.313700390" Nov 01 20:36:23.517658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:23.515902 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:23.595810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:23.595775 5444 patch_prober.go:29] interesting pod/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:36:23.595964 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:23.595824 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:36:28.169775 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.169745 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:36:28.280707 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.280682 5444 generic.go:356] "Generic (PLEG): container finished" podID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerID="5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9" exitCode=0 Nov 01 20:36:28.280833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.280723 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" Nov 01 20:36:28.280833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.280723 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" event={"ID":"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db","Type":"ContainerDied","Data":"5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9"} Nov 01 20:36:28.280960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.280876 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx" event={"ID":"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db","Type":"ContainerDied","Data":"1bcb9b37638a0145c435a373abf0ad98d3ff3f2321b4cd7ff5381cdef211d3f4"} Nov 01 20:36:28.280960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.280913 5444 scope.go:122] "RemoveContainer" containerID="5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9" Nov 01 20:36:28.289135 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.289119 5444 scope.go:122] "RemoveContainer" containerID="5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9" Nov 01 20:36:28.289376 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:36:28.289358 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9\": container with ID starting with 5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9 not found: ID does not exist" containerID="5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9" Nov 01 20:36:28.289423 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.289385 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9"} err="failed to get container status \"5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9\": rpc error: code = NotFound desc = could not find container \"5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9\": container with ID starting with 5b7877ee41709812a8814f3d3ada57d1e8ec5e1dcb0900cb230592ad043a16f9 not found: ID does not exist" Nov 01 20:36:28.323665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323625 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.323748 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323689 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.323748 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323718 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.323748 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323742 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.323913 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323766 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.323913 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323866 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.324020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323944 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.324020 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323979 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.324198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324035 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket\") pod \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\" (UID: \"a67dbfe8-ae03-4b42-a676-b38bcf4eb2db\") " Nov 01 20:36:28.324253 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.323872 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:28.324253 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324153 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:28.324253 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324176 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:28.324253 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324186 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:36:28.324482 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324449 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:28.324539 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324515 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.324539 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324529 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.324665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324539 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.324665 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.324549 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.325892 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.325864 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:36:28.325982 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.325962 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:28.326249 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.326234 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "kube-api-access-q9k2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:36:28.326343 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.326324 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo" pod "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" (UID: "a67dbfe8-ae03-4b42-a676-b38bcf4eb2db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:36:28.425517 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.425491 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.425517 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.425515 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.425667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.425525 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.425667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.425533 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.425667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.425550 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9k2s\" (UniqueName: \"kubernetes.io/projected/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db-kube-api-access-q9k2s\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:36:28.700440 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.700410 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx"] Nov 01 20:36:28.750189 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:28.750164 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-b7xvx"] Nov 01 20:36:29.158480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:29.158451 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" path="/var/lib/kubelet/pods/a67dbfe8-ae03-4b42-a676-b38bcf4eb2db/volumes" Nov 01 20:36:30.034633 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:36:30.034597 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:36:32.517743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:32.517699 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:32.714523 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:32.714221 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:33.330884 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:33.330853 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:36:41.751524 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.751493 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc"] Nov 01 20:36:41.751953 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.751941 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" Nov 01 20:36:41.752004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.751955 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" Nov 01 20:36:41.752054 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.752044 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67dbfe8-ae03-4b42-a676-b38bcf4eb2db" containerName="istio-proxy" Nov 01 20:36:41.755192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.755166 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.796027 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.795998 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.796156 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.796038 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.796210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.796176 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cdj\" (UniqueName: \"kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.796258 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.796211 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.806905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.806883 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-012e1c-serving-cert\"" Nov 01 20:36:41.858223 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.858197 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc"] Nov 01 20:36:41.897612 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.897588 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cdj\" (UniqueName: \"kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.897732 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.897624 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.897732 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.897707 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.897817 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.897731 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.897856 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:36:41.897830 5444 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-serving-cert: secret "maas-api-e2e-api-a-012e1c-serving-cert" not found Nov 01 20:36:41.897905 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:36:41.897899 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls podName:1e21b4f7-52b9-40e6-bc00-c5ba1158d641 nodeName:}" failed. No retries permitted until 2026-11-01 20:36:42.397879745 +0000 UTC m=+2003.342717596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls") pod "maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" (UID: "1e21b4f7-52b9-40e6-bc00-c5ba1158d641") : secret "maas-api-e2e-api-a-012e1c-serving-cert" not found Nov 01 20:36:41.898494 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.898472 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.900429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.900410 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:41.954753 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:41.954726 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cdj\" (UniqueName: \"kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:42.403528 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:42.403495 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:42.405834 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:42.405811 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\") pod \"maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:42.667807 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:42.667745 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:43.039576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:43.039550 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc"] Nov 01 20:36:43.041735 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:36:43.041710 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-37d759fcdd455c676877ad6c47d067c53acd59a952958a5cf679887c9a3b7b7c WatchSource:0}: Error finding container 37d759fcdd455c676877ad6c47d067c53acd59a952958a5cf679887c9a3b7b7c: Status 404 returned error can't find the container with id 37d759fcdd455c676877ad6c47d067c53acd59a952958a5cf679887c9a3b7b7c Nov 01 20:36:43.043206 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:43.043190 5444 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:36:43.380606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:43.380542 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" event={"ID":"1e21b4f7-52b9-40e6-bc00-c5ba1158d641","Type":"ContainerStarted","Data":"37d759fcdd455c676877ad6c47d067c53acd59a952958a5cf679887c9a3b7b7c"} Nov 01 20:36:44.387119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:44.387078 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" event={"ID":"1e21b4f7-52b9-40e6-bc00-c5ba1158d641","Type":"ContainerStarted","Data":"9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a"} Nov 01 20:36:44.387488 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:44.387145 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:44.500262 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:44.500217 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" podStartSLOduration=2.655041355 podStartE2EDuration="3.500202229s" totalImagesPullingTime="845.160874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:36:44.499530884 +0000 UTC m=+2005.444368756" watchObservedRunningTime="2026-11-01 20:36:44.500202229 +0000 UTC m=+2005.445040137" Nov 01 20:36:50.397250 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:50.397218 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:36:59.305240 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:36:59.305207 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8"] Nov 01 20:37:00.035179 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:37:00.035146 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:37:14.324450 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.324416 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" podUID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" containerName="main" containerID="cri-o://7201c7047510cd74cc46ed373d9295e104736308a2caaf34bc5ecc1435e3e4ca" gracePeriod=45 Nov 01 20:37:14.328001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.327970 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" podUID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": EOF" Nov 01 20:37:14.537340 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.537308 5444 generic.go:356] "Generic (PLEG): container finished" podID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" containerID="7201c7047510cd74cc46ed373d9295e104736308a2caaf34bc5ecc1435e3e4ca" exitCode=0 Nov 01 20:37:14.537482 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.537396 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" event={"ID":"5da6988b-be1e-4766-b8d8-f5aea4a72a5d","Type":"ContainerDied","Data":"7201c7047510cd74cc46ed373d9295e104736308a2caaf34bc5ecc1435e3e4ca"} Nov 01 20:37:14.577724 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.577676 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:37:14.737179 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737148 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79\" (UniqueName: \"kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79\") pod \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " Nov 01 20:37:14.737331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737242 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache\") pod \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " Nov 01 20:37:14.737331 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737273 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home\") pod \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " Nov 01 20:37:14.737446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737348 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir\") pod \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " Nov 01 20:37:14.737446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737368 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm\") pod \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " Nov 01 20:37:14.737446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737415 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs\") pod \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\" (UID: \"5da6988b-be1e-4766-b8d8-f5aea4a72a5d\") " Nov 01 20:37:14.737605 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737529 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache" pod "5da6988b-be1e-4766-b8d8-f5aea4a72a5d" (UID: "5da6988b-be1e-4766-b8d8-f5aea4a72a5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:14.737702 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737612 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home" pod "5da6988b-be1e-4766-b8d8-f5aea4a72a5d" (UID: "5da6988b-be1e-4766-b8d8-f5aea4a72a5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:14.737828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737802 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir" pod "5da6988b-be1e-4766-b8d8-f5aea4a72a5d" (UID: "5da6988b-be1e-4766-b8d8-f5aea4a72a5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:14.737906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737888 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:14.737952 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.737910 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:14.739415 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.739393 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79" pod "5da6988b-be1e-4766-b8d8-f5aea4a72a5d" (UID: "5da6988b-be1e-4766-b8d8-f5aea4a72a5d"). InnerVolumeSpecName "kube-api-access-zzz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:14.739415 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.739408 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs" pod "5da6988b-be1e-4766-b8d8-f5aea4a72a5d" (UID: "5da6988b-be1e-4766-b8d8-f5aea4a72a5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:37:14.739583 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.739413 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm" pod "5da6988b-be1e-4766-b8d8-f5aea4a72a5d" (UID: "5da6988b-be1e-4766-b8d8-f5aea4a72a5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:37:14.838721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.838669 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:14.838721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.838692 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:14.838721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.838701 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:14.838721 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:14.838710 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzz79\" (UniqueName: \"kubernetes.io/projected/5da6988b-be1e-4766-b8d8-f5aea4a72a5d-kube-api-access-zzz79\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:37:15.544737 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:15.544699 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" event={"ID":"5da6988b-be1e-4766-b8d8-f5aea4a72a5d","Type":"ContainerDied","Data":"a99604fa0d41fad0a654949fabc1b1330c57a844dd3e45d6666dc1fbc63e2997"} Nov 01 20:37:15.547037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:15.544755 5444 scope.go:122] "RemoveContainer" containerID="7201c7047510cd74cc46ed373d9295e104736308a2caaf34bc5ecc1435e3e4ca" Nov 01 20:37:15.547037 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:15.544758 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8" Nov 01 20:37:15.709387 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:15.709359 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8"] Nov 01 20:37:15.758465 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:15.758441 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-912d17/noten-model-912d17-kserve-76bddc988f-2mwl8"] Nov 01 20:37:17.158394 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:17.158362 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" path="/var/lib/kubelet/pods/5da6988b-be1e-4766-b8d8-f5aea4a72a5d/volumes" Nov 01 20:37:30.422213 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:37:30.422173 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:37:46.761649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:46.761604 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8"] Nov 01 20:37:46.811380 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:37:46.811347 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp"] Nov 01 20:38:00.034091 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:00.034016 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:38:01.781358 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:01.781302 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" podUID="70018ba0-cddf-45fd-90c2-843278f3bfd7" containerName="main" containerID="cri-o://7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766" gracePeriod=45 Nov 01 20:38:01.829621 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:01.829588 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" podUID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" containerName="main" containerID="cri-o://23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6" gracePeriod=45 Nov 01 20:38:02.101832 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.101807 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:38:02.104998 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.104976 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:38:02.190888 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.190857 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir\") pod \"70018ba0-cddf-45fd-90c2-843278f3bfd7\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " Nov 01 20:38:02.191023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.190900 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm\") pod \"70018ba0-cddf-45fd-90c2-843278f3bfd7\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " Nov 01 20:38:02.191023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.190988 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home\") pod \"70018ba0-cddf-45fd-90c2-843278f3bfd7\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " Nov 01 20:38:02.191150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191022 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home\") pod \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " Nov 01 20:38:02.191150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191055 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache\") pod \"70018ba0-cddf-45fd-90c2-843278f3bfd7\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " Nov 01 20:38:02.191150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191111 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs\" (UniqueName: \"kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs\") pod \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " Nov 01 20:38:02.191150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191134 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir" pod "70018ba0-cddf-45fd-90c2-843278f3bfd7" (UID: "70018ba0-cddf-45fd-90c2-843278f3bfd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.191421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191178 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm\") pod \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " Nov 01 20:38:02.191421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191247 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc\" (UniqueName: \"kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc\") pod \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " Nov 01 20:38:02.191421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191277 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir\") pod \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " Nov 01 20:38:02.191421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191305 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs\") pod \"70018ba0-cddf-45fd-90c2-843278f3bfd7\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " Nov 01 20:38:02.191421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191346 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home" pod "53fe30f6-048b-4ddb-8c8e-33c4ca486e36" (UID: "53fe30f6-048b-4ddb-8c8e-33c4ca486e36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.191421 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191390 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache\") pod \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\" (UID: \"53fe30f6-048b-4ddb-8c8e-33c4ca486e36\") " Nov 01 20:38:02.191788 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191426 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6\" (UniqueName: \"kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6\") pod \"70018ba0-cddf-45fd-90c2-843278f3bfd7\" (UID: \"70018ba0-cddf-45fd-90c2-843278f3bfd7\") " Nov 01 20:38:02.191788 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191541 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home" pod "70018ba0-cddf-45fd-90c2-843278f3bfd7" (UID: "70018ba0-cddf-45fd-90c2-843278f3bfd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.191788 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191742 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache" pod "70018ba0-cddf-45fd-90c2-843278f3bfd7" (UID: "70018ba0-cddf-45fd-90c2-843278f3bfd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.191951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191852 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.191951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191871 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.191951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191884 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.191951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.191899 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.192203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.192129 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache" pod "53fe30f6-048b-4ddb-8c8e-33c4ca486e36" (UID: "53fe30f6-048b-4ddb-8c8e-33c4ca486e36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.192626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.192595 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir" pod "53fe30f6-048b-4ddb-8c8e-33c4ca486e36" (UID: "53fe30f6-048b-4ddb-8c8e-33c4ca486e36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.195826 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.195793 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm" pod "70018ba0-cddf-45fd-90c2-843278f3bfd7" (UID: "70018ba0-cddf-45fd-90c2-843278f3bfd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.195931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.195889 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6" pod "70018ba0-cddf-45fd-90c2-843278f3bfd7" (UID: "70018ba0-cddf-45fd-90c2-843278f3bfd7"). InnerVolumeSpecName "kube-api-access-vj2w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:02.196088 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.196068 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc" pod "53fe30f6-048b-4ddb-8c8e-33c4ca486e36" (UID: "53fe30f6-048b-4ddb-8c8e-33c4ca486e36"). InnerVolumeSpecName "kube-api-access-r25rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:02.196088 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.196077 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm" pod "53fe30f6-048b-4ddb-8c8e-33c4ca486e36" (UID: "53fe30f6-048b-4ddb-8c8e-33c4ca486e36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:02.196270 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.196240 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs" pod "53fe30f6-048b-4ddb-8c8e-33c4ca486e36" (UID: "53fe30f6-048b-4ddb-8c8e-33c4ca486e36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:38:02.196479 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.196463 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs" pod "70018ba0-cddf-45fd-90c2-843278f3bfd7" (UID: "70018ba0-cddf-45fd-90c2-843278f3bfd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:38:02.293119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293084 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293119 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293118 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2w6\" (UniqueName: \"kubernetes.io/projected/70018ba0-cddf-45fd-90c2-843278f3bfd7-kube-api-access-vj2w6\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293134 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70018ba0-cddf-45fd-90c2-843278f3bfd7-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293149 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293162 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293178 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25rc\" (UniqueName: \"kubernetes.io/projected/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-kube-api-access-r25rc\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293194 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fe30f6-048b-4ddb-8c8e-33c4ca486e36-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.293363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.293208 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70018ba0-cddf-45fd-90c2-843278f3bfd7-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:02.893467 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.893434 5444 generic.go:356] "Generic (PLEG): container finished" podID="70018ba0-cddf-45fd-90c2-843278f3bfd7" containerID="7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766" exitCode=0 Nov 01 20:38:02.893909 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.893488 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" Nov 01 20:38:02.893909 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.893519 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" event={"ID":"70018ba0-cddf-45fd-90c2-843278f3bfd7","Type":"ContainerDied","Data":"7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766"} Nov 01 20:38:02.893909 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.893558 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8" event={"ID":"70018ba0-cddf-45fd-90c2-843278f3bfd7","Type":"ContainerDied","Data":"a34128915bde976008c2a24f4820a8a3f65a7f0a04e23e4d085d7a969e1453b6"} Nov 01 20:38:02.893909 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.893576 5444 scope.go:122] "RemoveContainer" containerID="7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766" Nov 01 20:38:02.894879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.894763 5444 generic.go:356] "Generic (PLEG): container finished" podID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" containerID="23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6" exitCode=0 Nov 01 20:38:02.894879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.894797 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" event={"ID":"53fe30f6-048b-4ddb-8c8e-33c4ca486e36","Type":"ContainerDied","Data":"23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6"} Nov 01 20:38:02.894879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.894841 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" event={"ID":"53fe30f6-048b-4ddb-8c8e-33c4ca486e36","Type":"ContainerDied","Data":"e31961455f753ee5b87753868089041dc81c19067b6ba69f9138c3261b7a6c85"} Nov 01 20:38:02.894879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.894809 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp" Nov 01 20:38:02.902733 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.902710 5444 scope.go:122] "RemoveContainer" containerID="7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766" Nov 01 20:38:02.903012 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:02.902984 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766\": container with ID starting with 7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766 not found: ID does not exist" containerID="7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766" Nov 01 20:38:02.903106 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.903021 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766"} err="failed to get container status \"7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766\": rpc error: code = NotFound desc = could not find container \"7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766\": container with ID starting with 7f8e779a6d601ff7f36af4cad9a27704e4da63de5d407dd4b0f6a01050322766 not found: ID does not exist" Nov 01 20:38:02.903106 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.903046 5444 scope.go:122] "RemoveContainer" containerID="23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6" Nov 01 20:38:02.910616 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.910600 5444 scope.go:122] "RemoveContainer" containerID="23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6" Nov 01 20:38:02.910933 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:02.910914 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6\": container with ID starting with 23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6 not found: ID does not exist" containerID="23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6" Nov 01 20:38:02.911002 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:02.910940 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6"} err="failed to get container status \"23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6\": rpc error: code = NotFound desc = could not find container \"23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6\": container with ID starting with 23ccebd6f67d7a5aca3a2cc3a60b5d46fb88f3dc75660d0fd5f51a3c7f63a0b6 not found: ID does not exist" Nov 01 20:38:03.056548 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:03.056482 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp"] Nov 01 20:38:03.105842 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:03.105819 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/simulator-w0-d204f4-kserve-69794c99f5-5kbrp"] Nov 01 20:38:03.158323 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:03.158299 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" path="/var/lib/kubelet/pods/53fe30f6-048b-4ddb-8c8e-33c4ca486e36/volumes" Nov 01 20:38:03.215092 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:03.215060 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8"] Nov 01 20:38:03.216519 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:03.216498 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d204f4/premium-w0-d204f4-kserve-6b9c6dd974-bv9m8"] Nov 01 20:38:05.157422 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:05.157390 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70018ba0-cddf-45fd-90c2-843278f3bfd7" path="/var/lib/kubelet/pods/70018ba0-cddf-45fd-90c2-843278f3bfd7/volumes" Nov 01 20:38:30.102113 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:30.102079 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:38:32.572008 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.571974 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt"] Nov 01 20:38:32.572418 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.572191 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" podUID="559d0cbf-8dd2-4ef7-b33b-056ae3643e12" containerName="maas-api" containerID="cri-o://8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955" gracePeriod=30 Nov 01 20:38:32.916173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.916148 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:38:32.972662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.972615 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca\") pod \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " Nov 01 20:38:32.972848 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.972829 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs\") pod \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " Nov 01 20:38:32.972892 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.972878 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p\" (UniqueName: \"kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p\") pod \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " Nov 01 20:38:32.972928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.972914 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\") pod \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\" (UID: \"559d0cbf-8dd2-4ef7-b33b-056ae3643e12\") " Nov 01 20:38:32.973023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.973004 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca" pod "559d0cbf-8dd2-4ef7-b33b-056ae3643e12" (UID: "559d0cbf-8dd2-4ef7-b33b-056ae3643e12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:32.973310 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.973287 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:32.974957 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.974935 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p" pod "559d0cbf-8dd2-4ef7-b33b-056ae3643e12" (UID: "559d0cbf-8dd2-4ef7-b33b-056ae3643e12"). InnerVolumeSpecName "kube-api-access-qs56p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:32.975316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.975289 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs" pod "559d0cbf-8dd2-4ef7-b33b-056ae3643e12" (UID: "559d0cbf-8dd2-4ef7-b33b-056ae3643e12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:38:32.975316 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:32.975299 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls" pod "559d0cbf-8dd2-4ef7-b33b-056ae3643e12" (UID: "559d0cbf-8dd2-4ef7-b33b-056ae3643e12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:38:33.074321 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.074284 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:33.074321 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.074314 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs56p\" (UniqueName: \"kubernetes.io/projected/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-kube-api-access-qs56p\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:33.074321 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.074329 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/559d0cbf-8dd2-4ef7-b33b-056ae3643e12-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:33.088305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.088267 5444 generic.go:356] "Generic (PLEG): container finished" podID="559d0cbf-8dd2-4ef7-b33b-056ae3643e12" containerID="8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955" exitCode=0 Nov 01 20:38:33.088462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.088327 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" Nov 01 20:38:33.088462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.088351 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" event={"ID":"559d0cbf-8dd2-4ef7-b33b-056ae3643e12","Type":"ContainerDied","Data":"8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955"} Nov 01 20:38:33.088462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.088408 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt" event={"ID":"559d0cbf-8dd2-4ef7-b33b-056ae3643e12","Type":"ContainerDied","Data":"9e1fbfd8aff62fde6e3af2f5045c94ff826781d7cc4b6e62e82b227a0c5ed592"} Nov 01 20:38:33.088462 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.088430 5444 scope.go:122] "RemoveContainer" containerID="8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955" Nov 01 20:38:33.105467 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.105447 5444 scope.go:122] "RemoveContainer" containerID="8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955" Nov 01 20:38:33.105760 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:33.105741 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955\": container with ID starting with 8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955 not found: ID does not exist" containerID="8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955" Nov 01 20:38:33.105828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.105770 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955"} err="failed to get container status \"8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955\": rpc error: code = NotFound desc = could not find container \"8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955\": container with ID starting with 8625c09869cf8653ce54036c35c48cfd5a3ed8880eb9fbf1ae1975ef853d6955 not found: ID does not exist" Nov 01 20:38:33.280230 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.280196 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt"] Nov 01 20:38:33.283527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:33.283506 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d204f4-cf4764b7f-dcgjt"] Nov 01 20:38:35.158223 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:35.158193 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559d0cbf-8dd2-4ef7-b33b-056ae3643e12" path="/var/lib/kubelet/pods/559d0cbf-8dd2-4ef7-b33b-056ae3643e12/volumes" Nov 01 20:38:39.075551 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.075520 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw"] Nov 01 20:38:39.075994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.075979 5444 cpu_manager.go:414] "containerMap: removing container" podUID="559d0cbf-8dd2-4ef7-b33b-056ae3643e12" containerName="maas-api" Nov 01 20:38:39.076038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.075995 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559d0cbf-8dd2-4ef7-b33b-056ae3643e12" containerName="maas-api" Nov 01 20:38:39.076038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076014 5444 cpu_manager.go:414] "containerMap: removing container" podUID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" containerName="main" Nov 01 20:38:39.076038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076020 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" containerName="main" Nov 01 20:38:39.076038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076033 5444 cpu_manager.go:414] "containerMap: removing container" podUID="70018ba0-cddf-45fd-90c2-843278f3bfd7" containerName="main" Nov 01 20:38:39.076038 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076038 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70018ba0-cddf-45fd-90c2-843278f3bfd7" containerName="main" Nov 01 20:38:39.076192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076049 5444 cpu_manager.go:414] "containerMap: removing container" podUID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" containerName="main" Nov 01 20:38:39.076192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076053 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" containerName="main" Nov 01 20:38:39.076192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076126 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="559d0cbf-8dd2-4ef7-b33b-056ae3643e12" containerName="maas-api" Nov 01 20:38:39.076192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076135 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da6988b-be1e-4766-b8d8-f5aea4a72a5d" containerName="main" Nov 01 20:38:39.076192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076142 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="70018ba0-cddf-45fd-90c2-843278f3bfd7" containerName="main" Nov 01 20:38:39.076192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.076149 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fe30f6-048b-4ddb-8c8e-33c4ca486e36" containerName="main" Nov 01 20:38:39.080317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.080300 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.146897 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.146856 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28m8q\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.146989 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147032 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147062 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147098 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147087 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147242 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147286 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147450 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147384 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.147493 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.147444 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.221220 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.221184 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw"] Nov 01 20:38:39.249004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.248957 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249004 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249008 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249122 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249179 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249201 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249304 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249379 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249395 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249585 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249457 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28m8q\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249677 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249585 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249677 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249586 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249809 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249688 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249871 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249817 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.249912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.249873 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.251448 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.251426 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.251558 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.251503 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.306805 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.306778 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.307127 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.307105 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28m8q\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q\") pod \"e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.392508 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.392421 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:39.827310 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.827222 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw"] Nov 01 20:38:39.830359 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:38:39.830330 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda416c61c_ea5f_42c0_99e3_1f47cb6bb194.slice/crio-6d0f5e459217462d0e277045b1d94794a3d4e49c9c2816acf6eea9dc68cbed94 WatchSource:0}: Error finding container 6d0f5e459217462d0e277045b1d94794a3d4e49c9c2816acf6eea9dc68cbed94: Status 404 returned error can't find the container with id 6d0f5e459217462d0e277045b1d94794a3d4e49c9c2816acf6eea9dc68cbed94 Nov 01 20:38:39.832728 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.832696 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:38:39.832791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.832764 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:38:39.832831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:39.832801 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:38:40.124533 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:40.124445 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" event={"ID":"a416c61c-ea5f-42c0-99e3-1f47cb6bb194","Type":"ContainerStarted","Data":"75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59"} Nov 01 20:38:40.124533 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:40.124510 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" event={"ID":"a416c61c-ea5f-42c0-99e3-1f47cb6bb194","Type":"ContainerStarted","Data":"6d0f5e459217462d0e277045b1d94794a3d4e49c9c2816acf6eea9dc68cbed94"} Nov 01 20:38:40.244426 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:40.244375 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" podStartSLOduration=1.244361313 podStartE2EDuration="1.244361313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:40.241562569 +0000 UTC m=+2121.186400437" watchObservedRunningTime="2026-11-01 20:38:40.244361313 +0000 UTC m=+2121.189199182" Nov 01 20:38:40.393014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:40.392925 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:40.394050 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:40.394026 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 20:38:40.394173 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:40.394070 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 20:38:41.392794 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.392758 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 20:38:41.393251 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.392812 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 20:38:41.496832 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.496794 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw"] Nov 01 20:38:41.497055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.497030 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" containerID="cri-o://bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245" gracePeriod=30 Nov 01 20:38:41.531407 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.531370 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj"] Nov 01 20:38:41.531649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.531609 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" containerID="cri-o://b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d" gracePeriod=30 Nov 01 20:38:41.556703 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:41.555701 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw"] Nov 01 20:38:42.136999 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:42.136956 5444 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" secret="" err="secret \"e2e-worker-w0-d204f4-openshift-default-dockercfg-n26pz\" not found" Nov 01 20:38:42.392790 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:42.392713 5444 patch_prober.go:29] interesting pod/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 01 20:38:42.393047 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:42.393020 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 01 20:38:43.142516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:43.142473 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" containerID="cri-o://75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59" gracePeriod=30 Nov 01 20:38:46.793717 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.793690 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:38:46.817912 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.817891 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:38:46.925236 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925212 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925253 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925286 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925318 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925383 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925346 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925394 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925449 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925501 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925550 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925577 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925614 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925607 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925618 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925686 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925725 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925778 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925813 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925856 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925878 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925906 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data\") pod \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\" (UID: \"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9\") " Nov 01 20:38:46.925976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.925949 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token\") pod \"e0b2eff4-709a-47a6-92aa-632d962238b7\" (UID: \"e0b2eff4-709a-47a6-92aa-632d962238b7\") " Nov 01 20:38:46.926443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926042 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.926443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926391 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:46.926443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926411 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:46.926443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926425 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:46.926672 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926453 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.926742 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926715 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.926996 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.926972 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.927268 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.927208 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.927623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.927390 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:46.927623 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.927439 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.928255 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.927997 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "kube-api-access-2gtdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:46.928255 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.928160 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:46.928420 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.928399 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:38:46.928533 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.928490 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.929360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.929330 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:38:46.929781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.929758 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:46.929987 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.929970 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token" pod "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" (UID: "1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:46.930205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.930190 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:46.930417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:46.930394 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8" pod "e0b2eff4-709a-47a6-92aa-632d962238b7" (UID: "e0b2eff4-709a-47a6-92aa-632d962238b7"). InnerVolumeSpecName "kube-api-access-6bmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:47.027629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027558 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b2eff4-709a-47a6-92aa-632d962238b7-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027584 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmc8\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-kube-api-access-6bmc8\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027595 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027603 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027614 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027622 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtdc\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-kube-api-access-2gtdc\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027630 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027662 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027671 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027679 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027687 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027696 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b2eff4-709a-47a6-92aa-632d962238b7-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027704 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b2eff4-709a-47a6-92aa-632d962238b7-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027712 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.027934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.027720 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:47.230633 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.230597 5444 generic.go:356] "Generic (PLEG): container finished" podID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerID="bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245" exitCode=0 Nov 01 20:38:47.230825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.230661 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" event={"ID":"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9","Type":"ContainerDied","Data":"bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245"} Nov 01 20:38:47.230825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.230676 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" Nov 01 20:38:47.230825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.230732 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw" event={"ID":"1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9","Type":"ContainerDied","Data":"7c427ea6b1fb75f3776714bc5d04aed94677c01a968ac0e5bc8a031b68603f55"} Nov 01 20:38:47.230825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.230767 5444 scope.go:122] "RemoveContainer" containerID="bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245" Nov 01 20:38:47.233858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.233834 5444 generic.go:356] "Generic (PLEG): container finished" podID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerID="b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d" exitCode=0 Nov 01 20:38:47.233966 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.233904 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" event={"ID":"e0b2eff4-709a-47a6-92aa-632d962238b7","Type":"ContainerDied","Data":"b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d"} Nov 01 20:38:47.233966 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.233945 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" event={"ID":"e0b2eff4-709a-47a6-92aa-632d962238b7","Type":"ContainerDied","Data":"cad00dc129aa9b2983359f307e09ab5d06edcdb9b349d6c4e578fc05cb5bcafe"} Nov 01 20:38:47.234054 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.234015 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj" Nov 01 20:38:47.241695 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.241673 5444 scope.go:122] "RemoveContainer" containerID="bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245" Nov 01 20:38:47.242013 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:47.241992 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245\": container with ID starting with bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245 not found: ID does not exist" containerID="bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245" Nov 01 20:38:47.242109 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.242026 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245"} err="failed to get container status \"bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245\": rpc error: code = NotFound desc = could not find container \"bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245\": container with ID starting with bb404ecf3cc71dc7f50d87b85c0eb66301f10e6b3a2c622479a19d23656e5245 not found: ID does not exist" Nov 01 20:38:47.242109 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.242049 5444 scope.go:122] "RemoveContainer" containerID="b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d" Nov 01 20:38:47.249767 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.249748 5444 scope.go:122] "RemoveContainer" containerID="b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d" Nov 01 20:38:47.250031 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:47.250012 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d\": container with ID starting with b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d not found: ID does not exist" containerID="b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d" Nov 01 20:38:47.250088 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.250036 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d"} err="failed to get container status \"b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d\": rpc error: code = NotFound desc = could not find container \"b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d\": container with ID starting with b6aae02098e5047ab7722a337911324d3147f45bee7bd1b29b5cf241f2a47e3d not found: ID does not exist" Nov 01 20:38:47.393475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.393404 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw"] Nov 01 20:38:47.395939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.395915 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-h42nw"] Nov 01 20:38:47.507453 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.507410 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj"] Nov 01 20:38:47.511899 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:47.511871 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-xzmkj"] Nov 01 20:38:49.158769 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:49.158738 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" path="/var/lib/kubelet/pods/1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9/volumes" Nov 01 20:38:49.159179 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:49.159165 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" path="/var/lib/kubelet/pods/e0b2eff4-709a-47a6-92aa-632d962238b7/volumes" Nov 01 20:38:53.512464 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.512436 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:53.695595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695510 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695567 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695607 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695657 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695702 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695748 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695779 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695889 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.695916 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.695912 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q\") pod \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\" (UID: \"a416c61c-ea5f-42c0-99e3-1f47cb6bb194\") " Nov 01 20:38:53.697203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.696602 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:53.697203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.696818 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:53.697203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.696992 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:53.697203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.697153 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:53.697835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.697812 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:38:53.699006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.698413 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:53.699006 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.698516 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "kube-api-access-28m8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:38:53.699168 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.699101 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:38:53.699333 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.699306 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy" pod "a416c61c-ea5f-42c0-99e3-1f47cb6bb194" (UID: "a416c61c-ea5f-42c0-99e3-1f47cb6bb194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:38:53.797327 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797298 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797327 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797325 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28m8q\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-kube-api-access-28m8q\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797337 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797346 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797355 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797363 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797370 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797379 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:53.797487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:53.797387 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a416c61c-ea5f-42c0-99e3-1f47cb6bb194-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:38:54.272946 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.272912 5444 generic.go:356] "Generic (PLEG): container finished" podID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerID="75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59" exitCode=0 Nov 01 20:38:54.273113 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.272963 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" event={"ID":"a416c61c-ea5f-42c0-99e3-1f47cb6bb194","Type":"ContainerDied","Data":"75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59"} Nov 01 20:38:54.273113 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.273005 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" event={"ID":"a416c61c-ea5f-42c0-99e3-1f47cb6bb194","Type":"ContainerDied","Data":"6d0f5e459217462d0e277045b1d94794a3d4e49c9c2816acf6eea9dc68cbed94"} Nov 01 20:38:54.273113 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.272999 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw" Nov 01 20:38:54.273113 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.273018 5444 scope.go:122] "RemoveContainer" containerID="75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59" Nov 01 20:38:54.285449 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.285402 5444 scope.go:122] "RemoveContainer" containerID="75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59" Nov 01 20:38:54.286017 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:38:54.285992 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59\": container with ID starting with 75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59 not found: ID does not exist" containerID="75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59" Nov 01 20:38:54.286136 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.286024 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59"} err="failed to get container status \"75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59\": rpc error: code = NotFound desc = could not find container \"75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59\": container with ID starting with 75a6ebd57295150cd08bb82e851cfd12ddaa32b53c8b2f319d7793137f18ae59 not found: ID does not exist" Nov 01 20:38:54.485487 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.485454 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw"] Nov 01 20:38:54.535411 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:54.535324 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d204f4-openshift-default-bb75c6977-8d4jw"] Nov 01 20:38:55.157164 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:38:55.157132 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" path="/var/lib/kubelet/pods/a416c61c-ea5f-42c0-99e3-1f47cb6bb194/volumes" Nov 01 20:39:00.035280 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:00.035246 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:39:04.147892 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:04.147858 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8"] Nov 01 20:39:04.148345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:04.148057 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" containerID="cri-o://4f076151a68602c3bcfc86faa73e705adfe9e210c0507edc521dda3c8cd7cef3" gracePeriod=30 Nov 01 20:39:09.349976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.349740 5444 generic.go:356] "Generic (PLEG): container finished" podID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerID="4f076151a68602c3bcfc86faa73e705adfe9e210c0507edc521dda3c8cd7cef3" exitCode=0 Nov 01 20:39:09.349976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.349846 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" event={"ID":"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6","Type":"ContainerDied","Data":"4f076151a68602c3bcfc86faa73e705adfe9e210c0507edc521dda3c8cd7cef3"} Nov 01 20:39:09.408599 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.408570 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:39:09.491683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491612 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.491783 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491685 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.491783 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491752 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.491886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491785 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.491886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491809 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.491886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491834 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.491886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491864 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.492094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491875 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:09.492094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491906 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:09.492094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491921 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.492094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.491959 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj\") pod \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\" (UID: \"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6\") " Nov 01 20:39:09.492094 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492078 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:09.492360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492113 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:09.492360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492208 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:09.492360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492336 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.492360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492355 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.492527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492369 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.492527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492384 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.492527 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.492398 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.494048 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.494019 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:09.494179 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.494161 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:09.494432 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.494417 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "kube-api-access-729rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:09.494491 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.494431 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo" pod "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" (UID: "7dc4a5c9-68f0-4237-999e-8567fd9ed9f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:09.593692 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.593669 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.593692 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.593691 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.593838 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.593700 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:09.593838 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:09.593710 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729rj\" (UniqueName: \"kubernetes.io/projected/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6-kube-api-access-729rj\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:10.355460 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:10.355429 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" Nov 01 20:39:10.355879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:10.355433 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8" event={"ID":"7dc4a5c9-68f0-4237-999e-8567fd9ed9f6","Type":"ContainerDied","Data":"40022cb07513649ac1d150c199a9976bcb0bed12f1a21c3133ffbbd26da9560f"} Nov 01 20:39:10.355879 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:10.355558 5444 scope.go:122] "RemoveContainer" containerID="4f076151a68602c3bcfc86faa73e705adfe9e210c0507edc521dda3c8cd7cef3" Nov 01 20:39:10.567055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:10.567025 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8"] Nov 01 20:39:10.616410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:10.616336 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-gl2n8"] Nov 01 20:39:11.157818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:11.157785 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" path="/var/lib/kubelet/pods/7dc4a5c9-68f0-4237-999e-8567fd9ed9f6/volumes" Nov 01 20:39:19.243235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.243203 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc"] Nov 01 20:39:19.243740 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.243418 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" podUID="1e21b4f7-52b9-40e6-bc00-c5ba1158d641" containerName="maas-api" containerID="cri-o://9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a" gracePeriod=30 Nov 01 20:39:19.397114 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.397084 5444 generic.go:356] "Generic (PLEG): container finished" podID="1e21b4f7-52b9-40e6-bc00-c5ba1158d641" containerID="9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a" exitCode=0 Nov 01 20:39:19.397265 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.397160 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" event={"ID":"1e21b4f7-52b9-40e6-bc00-c5ba1158d641","Type":"ContainerDied","Data":"9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a"} Nov 01 20:39:19.541977 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.541946 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:39:19.701239 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.701206 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj\" (UniqueName: \"kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj\") pod \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " Nov 01 20:39:19.701433 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.701255 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca\") pod \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " Nov 01 20:39:19.701433 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.701315 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs\") pod \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " Nov 01 20:39:19.701433 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.701342 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\") pod \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\" (UID: \"1e21b4f7-52b9-40e6-bc00-c5ba1158d641\") " Nov 01 20:39:19.701716 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.701684 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca" pod "1e21b4f7-52b9-40e6-bc00-c5ba1158d641" (UID: "1e21b4f7-52b9-40e6-bc00-c5ba1158d641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:19.703368 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.703342 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj" pod "1e21b4f7-52b9-40e6-bc00-c5ba1158d641" (UID: "1e21b4f7-52b9-40e6-bc00-c5ba1158d641"). InnerVolumeSpecName "kube-api-access-v4cdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:19.703907 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.703879 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls" pod "1e21b4f7-52b9-40e6-bc00-c5ba1158d641" (UID: "1e21b4f7-52b9-40e6-bc00-c5ba1158d641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:39:19.703973 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.703919 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs" pod "1e21b4f7-52b9-40e6-bc00-c5ba1158d641" (UID: "1e21b4f7-52b9-40e6-bc00-c5ba1158d641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:39:19.802787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.802656 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4cdj\" (UniqueName: \"kubernetes.io/projected/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-kube-api-access-v4cdj\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:19.802787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.802697 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:19.802787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.802717 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:19.802787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:19.802732 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e21b4f7-52b9-40e6-bc00-c5ba1158d641-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:20.403697 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:20.403666 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" Nov 01 20:39:20.403697 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:20.403683 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc" event={"ID":"1e21b4f7-52b9-40e6-bc00-c5ba1158d641","Type":"ContainerDied","Data":"37d759fcdd455c676877ad6c47d067c53acd59a952958a5cf679887c9a3b7b7c"} Nov 01 20:39:20.404212 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:20.403765 5444 scope.go:122] "RemoveContainer" containerID="9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a" Nov 01 20:39:20.565293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:20.565270 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc"] Nov 01 20:39:20.566223 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:20.566200 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:20.614507 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:20.614483 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-012e1c-54db5ddd65-9qnnc"] Nov 01 20:39:21.158045 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:21.158014 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e21b4f7-52b9-40e6-bc00-c5ba1158d641" path="/var/lib/kubelet/pods/1e21b4f7-52b9-40e6-bc00-c5ba1158d641/volumes" Nov 01 20:39:28.295401 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:28.295369 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:30.036915 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:30.036877 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:39:31.516252 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:31.516207 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:31.516686 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:31.516436 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:35.643870 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:35.643839 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:37.157762 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:37.157735 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697"] Nov 01 20:39:37.158148 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:37.157916 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" containerID="cri-o://a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb" gracePeriod=30 Nov 01 20:39:37.202586 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:37.202555 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf"] Nov 01 20:39:37.202835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:37.202796 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" containerID="cri-o://bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb" gracePeriod=30 Nov 01 20:39:38.306256 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:38.306214 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:40.391868 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:40.391834 5444 patch_prober.go:29] interesting pod/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:39:40.392237 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:40.391887 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:39:42.457576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.457547 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:39:42.486942 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.486923 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:39:42.489122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489103 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.489203 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489186 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.489254 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489215 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.489297 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489259 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.489345 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489324 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.489516 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489493 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.489654 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489617 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:42.489654 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489629 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.489757 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.489614 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.491015 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.490994 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:42.491150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.491132 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:42.543184 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.543163 5444 generic.go:356] "Generic (PLEG): container finished" podID="a9c1328b-d892-4179-898b-22ea94729d04" containerID="a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb" exitCode=0 Nov 01 20:39:42.543283 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.543207 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" Nov 01 20:39:42.543283 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.543256 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" event={"ID":"a9c1328b-d892-4179-898b-22ea94729d04","Type":"ContainerDied","Data":"a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb"} Nov 01 20:39:42.543377 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.543343 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697" event={"ID":"a9c1328b-d892-4179-898b-22ea94729d04","Type":"ContainerDied","Data":"1762853e37efed3cbed7b0997526ced4c63d95e8c98fd17109882f40f4d57adc"} Nov 01 20:39:42.543377 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.543367 5444 scope.go:122] "RemoveContainer" containerID="a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb" Nov 01 20:39:42.544581 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.544561 5444 generic.go:356] "Generic (PLEG): container finished" podID="6deeedf8-ee4b-4483-895d-30a4659da793" containerID="bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb" exitCode=0 Nov 01 20:39:42.544683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.544598 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" Nov 01 20:39:42.544683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.544619 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" event={"ID":"6deeedf8-ee4b-4483-895d-30a4659da793","Type":"ContainerDied","Data":"bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb"} Nov 01 20:39:42.544683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.544672 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf" event={"ID":"6deeedf8-ee4b-4483-895d-30a4659da793","Type":"ContainerDied","Data":"bf2f7fd87f9796416f9cb01d64ffb40d4c59e72da5721462a7ee49e159c612b9"} Nov 01 20:39:42.551506 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.551489 5444 scope.go:122] "RemoveContainer" containerID="a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb" Nov 01 20:39:42.551793 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:42.551776 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb\": container with ID starting with a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb not found: ID does not exist" containerID="a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb" Nov 01 20:39:42.551850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.551799 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb"} err="failed to get container status \"a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb\": rpc error: code = NotFound desc = could not find container \"a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb\": container with ID starting with a346ca5e22c8de9188c4dd8ca687c070cc898c7786a546468d6cc57081b623cb not found: ID does not exist" Nov 01 20:39:42.551850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.551825 5444 scope.go:122] "RemoveContainer" containerID="bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb" Nov 01 20:39:42.559306 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.559284 5444 scope.go:122] "RemoveContainer" containerID="bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb" Nov 01 20:39:42.559571 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:42.559554 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb\": container with ID starting with bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb not found: ID does not exist" containerID="bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb" Nov 01 20:39:42.559662 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.559577 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb"} err="failed to get container status \"bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb\": rpc error: code = NotFound desc = could not find container \"bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb\": container with ID starting with bec695117857ac2a2ea446debbeadbb2e88b73e5579fc4acf3a5b36ac8664efb not found: ID does not exist" Nov 01 20:39:42.590853 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.590834 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.590972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.590865 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.590972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.590896 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.590972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.590927 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.591143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591052 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.591143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591095 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.591143 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591128 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.591310 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591154 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.591310 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591242 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.591310 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591280 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy\") pod \"a9c1328b-d892-4179-898b-22ea94729d04\" (UID: \"a9c1328b-d892-4179-898b-22ea94729d04\") " Nov 01 20:39:42.591480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591335 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.591480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591341 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.591480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591371 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.591480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591402 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.591480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591429 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy\") pod \"6deeedf8-ee4b-4483-895d-30a4659da793\" (UID: \"6deeedf8-ee4b-4483-895d-30a4659da793\") " Nov 01 20:39:42.591749 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591520 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.591825 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591801 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c1328b-d892-4179-898b-22ea94729d04-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.591885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591827 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c1328b-d892-4179-898b-22ea94729d04-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.591885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591843 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.591885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591858 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.591885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591872 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.592275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.591887 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.592275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.592115 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.592275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.592122 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.592275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.592121 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.592275 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.592135 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.592486 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.592419 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:42.593450 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.593417 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "kube-api-access-rv4zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:42.594122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.594088 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:42.594396 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.594373 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:42.594475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.594374 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.594475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.594456 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy" pod "6deeedf8-ee4b-4483-895d-30a4659da793" (UID: "6deeedf8-ee4b-4483-895d-30a4659da793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:42.594784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.594766 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx" pod "a9c1328b-d892-4179-898b-22ea94729d04" (UID: "a9c1328b-d892-4179-898b-22ea94729d04"). InnerVolumeSpecName "kube-api-access-frclx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:42.692781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692729 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.692781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692749 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c1328b-d892-4179-898b-22ea94729d04-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.692781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692759 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.692781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692767 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.692781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692776 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deeedf8-ee4b-4483-895d-30a4659da793-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.692781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692784 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deeedf8-ee4b-4483-895d-30a4659da793-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.693039 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692793 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv4zb\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-kube-api-access-rv4zb\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.693039 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692802 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deeedf8-ee4b-4483-895d-30a4659da793-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.693039 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692811 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frclx\" (UniqueName: \"kubernetes.io/projected/a9c1328b-d892-4179-898b-22ea94729d04-kube-api-access-frclx\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.693039 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.692819 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deeedf8-ee4b-4483-895d-30a4659da793-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:42.964945 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.964871 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf"] Nov 01 20:39:42.969324 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:42.969297 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-zm7hf"] Nov 01 20:39:43.076900 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:43.076859 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697"] Nov 01 20:39:43.082048 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:43.082025 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-j4697"] Nov 01 20:39:43.157653 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:43.157615 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" path="/var/lib/kubelet/pods/6deeedf8-ee4b-4483-895d-30a4659da793/volumes" Nov 01 20:39:43.158068 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:43.158055 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c1328b-d892-4179-898b-22ea94729d04" path="/var/lib/kubelet/pods/a9c1328b-d892-4179-898b-22ea94729d04/volumes" Nov 01 20:39:44.621653 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:44.621603 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz"] Nov 01 20:39:44.670980 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:44.670948 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk"] Nov 01 20:39:47.627286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:47.627255 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn"] Nov 01 20:39:47.627743 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:47.627440 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" containerID="cri-o://85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b" gracePeriod=30 Nov 01 20:39:47.683914 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:47.683882 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z"] Nov 01 20:39:47.684205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:47.684183 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" containerID="cri-o://9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325" gracePeriod=30 Nov 01 20:39:48.355196 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:48.355164 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:49.150785 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:49.150744 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:39:49.151235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:49.150807 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:39:50.571797 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:50.571764 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:52.205658 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.205615 5444 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:39:52.206093 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.205686 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:39:52.931910 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.931881 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:39:52.955833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.955810 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:39:52.986835 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.986808 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.986921 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.986862 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.986921 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.986907 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987018 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.986932 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987018 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.986959 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987018 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.986995 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987159 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987057 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:52.987159 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987065 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987181 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987229 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:52.987261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987243 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs\") pod \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\" (UID: \"aec99b9b-1bfa-4049-8d7a-b9e22107165e\") " Nov 01 20:39:52.987455 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987435 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:52.987511 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987430 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:52.987670 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987615 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:52.987791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987776 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:52.987858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987798 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:52.987858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987814 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:52.987858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987828 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:52.987858 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.987842 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:52.989205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.989175 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "kube-api-access-klblb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:52.989289 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.989218 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:52.989289 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.989220 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:52.989437 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:52.989420 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy" pod "aec99b9b-1bfa-4049-8d7a-b9e22107165e" (UID: "aec99b9b-1bfa-4049-8d7a-b9e22107165e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:53.088577 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088544 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.088577 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088581 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.088800 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088616 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.088800 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088658 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.088800 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088724 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.088800 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088761 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.089001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088821 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.089001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088856 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.089001 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.088913 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket\") pod \"2d4c55be-d563-4731-bb10-9cabb14c0c73\" (UID: \"2d4c55be-d563-4731-bb10-9cabb14c0c73\") " Nov 01 20:39:53.089159 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089062 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:53.089213 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089161 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:39:53.089295 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089277 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:53.089373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089327 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:53.089425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089405 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:53.089497 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089481 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089536 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089504 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089536 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089519 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4c55be-d563-4731-bb10-9cabb14c0c73-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089535 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klblb\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-kube-api-access-klblb\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089550 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089566 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089578 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.089617 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.089592 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aec99b9b-1bfa-4049-8d7a-b9e22107165e-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.090953 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.090933 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "kube-api-access-9mxpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:53.091028 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.090979 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:39:53.091242 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.091222 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:39:53.091281 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.091220 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo" pod "2d4c55be-d563-4731-bb10-9cabb14c0c73" (UID: "2d4c55be-d563-4731-bb10-9cabb14c0c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:39:53.190569 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.190536 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.190569 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.190565 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.190772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.190577 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.190772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.190612 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxpd\" (UniqueName: \"kubernetes.io/projected/2d4c55be-d563-4731-bb10-9cabb14c0c73-kube-api-access-9mxpd\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.190772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.190627 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4c55be-d563-4731-bb10-9cabb14c0c73-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:39:53.603047 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.603010 5444 generic.go:356] "Generic (PLEG): container finished" podID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerID="9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325" exitCode=0 Nov 01 20:39:53.603529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.603085 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" Nov 01 20:39:53.603529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.603097 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" event={"ID":"2d4c55be-d563-4731-bb10-9cabb14c0c73","Type":"ContainerDied","Data":"9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325"} Nov 01 20:39:53.603529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.603157 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z" event={"ID":"2d4c55be-d563-4731-bb10-9cabb14c0c73","Type":"ContainerDied","Data":"a85790fd4eddd3c6dacfc19868d758cfae8da40b4ef1fafc7e49d8257e2422ae"} Nov 01 20:39:53.603529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.603187 5444 scope.go:122] "RemoveContainer" containerID="9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325" Nov 01 20:39:53.604429 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.604407 5444 generic.go:356] "Generic (PLEG): container finished" podID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerID="85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b" exitCode=0 Nov 01 20:39:53.604526 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.604440 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" event={"ID":"aec99b9b-1bfa-4049-8d7a-b9e22107165e","Type":"ContainerDied","Data":"85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b"} Nov 01 20:39:53.604526 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.604478 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" Nov 01 20:39:53.604526 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.604493 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn" event={"ID":"aec99b9b-1bfa-4049-8d7a-b9e22107165e","Type":"ContainerDied","Data":"1eb48c9ac76109e442def300101b473a59faf4654095ac5127ca8da0b736d877"} Nov 01 20:39:53.612128 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.612107 5444 scope.go:122] "RemoveContainer" containerID="9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325" Nov 01 20:39:53.612442 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:53.612411 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325\": container with ID starting with 9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325 not found: ID does not exist" containerID="9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325" Nov 01 20:39:53.612505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.612451 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325"} err="failed to get container status \"9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325\": rpc error: code = NotFound desc = could not find container \"9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325\": container with ID starting with 9dab4efedbdd54364c13cb61edc3927ecf305573856319017f3d61ec35977325 not found: ID does not exist" Nov 01 20:39:53.612505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.612469 5444 scope.go:122] "RemoveContainer" containerID="85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b" Nov 01 20:39:53.620625 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.620606 5444 scope.go:122] "RemoveContainer" containerID="85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b" Nov 01 20:39:53.620935 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:53.620917 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b\": container with ID starting with 85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b not found: ID does not exist" containerID="85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b" Nov 01 20:39:53.620999 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.620944 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b"} err="failed to get container status \"85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b\": rpc error: code = NotFound desc = could not find container \"85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b\": container with ID starting with 85e20a376a80a762b35f127f0e21ed51034f9add994797f33aa82889a943ff4b not found: ID does not exist" Nov 01 20:39:53.769833 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.769803 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z"] Nov 01 20:39:53.826683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.826292 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-8dm4z"] Nov 01 20:39:53.885126 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.885045 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn"] Nov 01 20:39:53.888308 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:53.888272 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc24d-openshift-default-6f964b956f-4j6bn"] Nov 01 20:39:55.158675 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:55.158630 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" path="/var/lib/kubelet/pods/2d4c55be-d563-4731-bb10-9cabb14c0c73/volumes" Nov 01 20:39:55.159245 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:55.159225 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" path="/var/lib/kubelet/pods/aec99b9b-1bfa-4049-8d7a-b9e22107165e/volumes" Nov 01 20:39:58.399527 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:39:58.399494 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:39:59.640900 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:59.640864 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerName="main" containerID="cri-o://d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5" gracePeriod=45 Nov 01 20:39:59.689893 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:59.689858 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerName="main" containerID="cri-o://2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c" gracePeriod=45 Nov 01 20:39:59.947502 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:59.947480 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:39:59.949645 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:39:59.949619 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:40:00.036183 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:00.036152 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:40:00.070476 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.070449 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn\" (UniqueName: \"kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn\") pod \"db17a633-8bf3-4d93-9ee2-ac49378aa623\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " Nov 01 20:40:00.070607 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.070489 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8\" (UniqueName: \"kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8\") pod \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " Nov 01 20:40:00.070607 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.070534 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache\") pod \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " Nov 01 20:40:00.070607 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.070587 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs\" (UniqueName: \"kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs\") pod \"db17a633-8bf3-4d93-9ee2-ac49378aa623\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " Nov 01 20:40:00.070902 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.070877 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache" pod "f4439e41-5d46-4b30-8b4e-d5d83f3e9520" (UID: "f4439e41-5d46-4b30-8b4e-d5d83f3e9520"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.071007 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.070987 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm\") pod \"db17a633-8bf3-4d93-9ee2-ac49378aa623\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " Nov 01 20:40:00.071066 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.071045 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs\") pod \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " Nov 01 20:40:00.071410 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.071386 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm\") pod \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " Nov 01 20:40:00.071786 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.071763 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home\") pod \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " Nov 01 20:40:00.072082 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072040 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home" pod "f4439e41-5d46-4b30-8b4e-d5d83f3e9520" (UID: "f4439e41-5d46-4b30-8b4e-d5d83f3e9520"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.072260 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072239 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir\") pod \"db17a633-8bf3-4d93-9ee2-ac49378aa623\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " Nov 01 20:40:00.072376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072317 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home\") pod \"db17a633-8bf3-4d93-9ee2-ac49378aa623\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " Nov 01 20:40:00.072376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072364 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache\") pod \"db17a633-8bf3-4d93-9ee2-ac49378aa623\" (UID: \"db17a633-8bf3-4d93-9ee2-ac49378aa623\") " Nov 01 20:40:00.072492 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072400 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir\") pod \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\" (UID: \"f4439e41-5d46-4b30-8b4e-d5d83f3e9520\") " Nov 01 20:40:00.072931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072891 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache" pod "db17a633-8bf3-4d93-9ee2-ac49378aa623" (UID: "db17a633-8bf3-4d93-9ee2-ac49378aa623"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.072931 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.072907 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home" pod "db17a633-8bf3-4d93-9ee2-ac49378aa623" (UID: "db17a633-8bf3-4d93-9ee2-ac49378aa623"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.073198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073046 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir" pod "db17a633-8bf3-4d93-9ee2-ac49378aa623" (UID: "db17a633-8bf3-4d93-9ee2-ac49378aa623"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.073198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073078 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8" pod "f4439e41-5d46-4b30-8b4e-d5d83f3e9520" (UID: "f4439e41-5d46-4b30-8b4e-d5d83f3e9520"). InnerVolumeSpecName "kube-api-access-xhgl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:00.073198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073159 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir" pod "f4439e41-5d46-4b30-8b4e-d5d83f3e9520" (UID: "f4439e41-5d46-4b30-8b4e-d5d83f3e9520"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.073198 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073178 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm" pod "db17a633-8bf3-4d93-9ee2-ac49378aa623" (UID: "db17a633-8bf3-4d93-9ee2-ac49378aa623"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.073595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073573 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073595 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073594 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073622 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073659 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgl8\" (UniqueName: \"kubernetes.io/projected/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-kube-api-access-xhgl8\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073674 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073695 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073718 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.073808 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073730 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db17a633-8bf3-4d93-9ee2-ac49378aa623-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.074212 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.073948 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn" pod "db17a633-8bf3-4d93-9ee2-ac49378aa623" (UID: "db17a633-8bf3-4d93-9ee2-ac49378aa623"). InnerVolumeSpecName "kube-api-access-4nhzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:00.074212 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.074132 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs" pod "db17a633-8bf3-4d93-9ee2-ac49378aa623" (UID: "db17a633-8bf3-4d93-9ee2-ac49378aa623"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:00.075513 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.075494 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs" pod "f4439e41-5d46-4b30-8b4e-d5d83f3e9520" (UID: "f4439e41-5d46-4b30-8b4e-d5d83f3e9520"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:00.075625 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.075607 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm" pod "f4439e41-5d46-4b30-8b4e-d5d83f3e9520" (UID: "f4439e41-5d46-4b30-8b4e-d5d83f3e9520"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:00.174561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.174479 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db17a633-8bf3-4d93-9ee2-ac49378aa623-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.174561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.174514 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.174561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.174524 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4439e41-5d46-4b30-8b4e-d5d83f3e9520-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.174561 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.174534 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nhzn\" (UniqueName: \"kubernetes.io/projected/db17a633-8bf3-4d93-9ee2-ac49378aa623-kube-api-access-4nhzn\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:00.661267 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.661237 5444 generic.go:356] "Generic (PLEG): container finished" podID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerID="d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5" exitCode=0 Nov 01 20:40:00.661698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.661277 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" Nov 01 20:40:00.661698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.661322 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" event={"ID":"f4439e41-5d46-4b30-8b4e-d5d83f3e9520","Type":"ContainerDied","Data":"d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5"} Nov 01 20:40:00.661698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.661364 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz" event={"ID":"f4439e41-5d46-4b30-8b4e-d5d83f3e9520","Type":"ContainerDied","Data":"6d6b1aa198a119257f2bf874c8c7286676a82b636d14a12544ca1fcfd554b1d5"} Nov 01 20:40:00.661698 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.661381 5444 scope.go:122] "RemoveContainer" containerID="d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5" Nov 01 20:40:00.662607 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.662586 5444 generic.go:356] "Generic (PLEG): container finished" podID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerID="2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c" exitCode=0 Nov 01 20:40:00.662690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.662627 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" event={"ID":"db17a633-8bf3-4d93-9ee2-ac49378aa623","Type":"ContainerDied","Data":"2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c"} Nov 01 20:40:00.662690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.662653 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" Nov 01 20:40:00.662690 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.662672 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk" event={"ID":"db17a633-8bf3-4d93-9ee2-ac49378aa623","Type":"ContainerDied","Data":"1bd032dcd7a40ccd5a72072b6c2e7dacd4a71550856114b8bf85e841a1501824"} Nov 01 20:40:00.670014 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.669995 5444 scope.go:122] "RemoveContainer" containerID="d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5" Nov 01 20:40:00.670341 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:00.670261 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5\": container with ID starting with d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5 not found: ID does not exist" containerID="d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5" Nov 01 20:40:00.670341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.670287 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5"} err="failed to get container status \"d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5\": rpc error: code = NotFound desc = could not find container \"d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5\": container with ID starting with d6b3cf8097ef3b4461a18ba3eefa8258f2a006ae3525e0cf1c03e1f0eaa3a3e5 not found: ID does not exist" Nov 01 20:40:00.670341 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.670308 5444 scope.go:122] "RemoveContainer" containerID="2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c" Nov 01 20:40:00.677857 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.677838 5444 scope.go:122] "RemoveContainer" containerID="2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c" Nov 01 20:40:00.678112 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:00.678093 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c\": container with ID starting with 2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c not found: ID does not exist" containerID="2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c" Nov 01 20:40:00.678168 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.678118 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c"} err="failed to get container status \"2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c\": rpc error: code = NotFound desc = could not find container \"2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c\": container with ID starting with 2e980b131e2d8c497445520ba856c8325bfe01ef2d57183564847b1fea7f5c1c not found: ID does not exist" Nov 01 20:40:00.823282 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.823245 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk"] Nov 01 20:40:00.827473 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.827445 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/simulator-w3-1b1134-kserve-644dc6d568-7kcdk"] Nov 01 20:40:00.934129 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.934050 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz"] Nov 01 20:40:00.937762 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:00.937733 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1b1134/premium-w3-1b1134-kserve-746746777c-6vfxz"] Nov 01 20:40:01.158113 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:01.158079 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" path="/var/lib/kubelet/pods/db17a633-8bf3-4d93-9ee2-ac49378aa623/volumes" Nov 01 20:40:01.158934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:01.158901 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" path="/var/lib/kubelet/pods/f4439e41-5d46-4b30-8b4e-d5d83f3e9520/volumes" Nov 01 20:40:05.668584 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:05.668549 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:40:08.411696 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:08.411660 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:40:18.454219 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:18.454188 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e21b4f7_52b9_40e6_bc00_c5ba1158d641.slice/crio-9bf326e71698995b92dbcb0be2fe802a84a3fe5c53298195b27caf3ca93f6c0a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:40:25.657285 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657250 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq"] Nov 01 20:40:25.657824 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657804 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" Nov 01 20:40:25.657881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657828 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" Nov 01 20:40:25.657881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657849 5444 cpu_manager.go:414] "containerMap: removing container" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" Nov 01 20:40:25.657881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657857 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" Nov 01 20:40:25.657881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657869 5444 cpu_manager.go:414] "containerMap: removing container" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerName="main" Nov 01 20:40:25.657881 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657877 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerName="main" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657892 5444 cpu_manager.go:414] "containerMap: removing container" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657900 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657911 5444 cpu_manager.go:414] "containerMap: removing container" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657918 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657946 5444 cpu_manager.go:414] "containerMap: removing container" podUID="1e21b4f7-52b9-40e6-bc00-c5ba1158d641" containerName="maas-api" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657953 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e21b4f7-52b9-40e6-bc00-c5ba1158d641" containerName="maas-api" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657967 5444 cpu_manager.go:414] "containerMap: removing container" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657974 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657986 5444 cpu_manager.go:414] "containerMap: removing container" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.657993 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658009 5444 cpu_manager.go:414] "containerMap: removing container" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerName="main" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658016 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerName="main" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658029 5444 cpu_manager.go:414] "containerMap: removing container" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" Nov 01 20:40:25.658034 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658036 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658057 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658064 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658168 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c1328b-d892-4179-898b-22ea94729d04" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658180 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4c55be-d563-4731-bb10-9cabb14c0c73" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658189 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4439e41-5d46-4b30-8b4e-d5d83f3e9520" containerName="main" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658197 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b2eff4-709a-47a6-92aa-632d962238b7" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658206 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec99b9b-1bfa-4049-8d7a-b9e22107165e" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658217 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e21b4f7-52b9-40e6-bc00-c5ba1158d641" containerName="maas-api" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658228 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="1795b94e-b55b-4d4c-96ef-6a20d0d6b5b9" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658238 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc4a5c9-68f0-4237-999e-8567fd9ed9f6" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658248 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="db17a633-8bf3-4d93-9ee2-ac49378aa623" containerName="main" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658258 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="6deeedf8-ee4b-4483-895d-30a4659da793" containerName="istio-proxy" Nov 01 20:40:25.658443 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.658270 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a416c61c-ea5f-42c0-99e3-1f47cb6bb194" containerName="istio-proxy" Nov 01 20:40:25.663238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.663214 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.704157 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.704128 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.704261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.704170 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkpq\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.704261 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.704232 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.704360 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.704268 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.713593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.713572 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq"] Nov 01 20:40:25.715353 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.715326 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-4bc05b0bda98\"" Nov 01 20:40:25.715353 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.715341 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-cd18c1528ab8\"" Nov 01 20:40:25.715547 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.715350 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 20:40:25.761994 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.761973 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-e385637602d5\"" Nov 01 20:40:25.762115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.762024 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-caf6c2f34a9e\"" Nov 01 20:40:25.762115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.762050 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-t8dl8\"" Nov 01 20:40:25.762115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.762055 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 20:40:25.762115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.762077 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 20:40:25.764408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.764394 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 20:40:25.805286 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.805261 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.805399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.805295 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkpq\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.805399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.805326 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.805399 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.805378 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.805968 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.805941 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.806105 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.806087 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.807917 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.807894 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.861774 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.861739 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkpq\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:25.986770 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:25.983306 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:26.372798 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:26.372759 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq"] Nov 01 20:40:26.377668 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:40:26.377628 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e0106a_3e12_4a6d_a721_2a11082f8e7b.slice/crio-3ac6dba1f7d8f5e6dc6b72e614afb9042e6a425e99392ba0aaac6d7a08aad4c1 WatchSource:0}: Error finding container 3ac6dba1f7d8f5e6dc6b72e614afb9042e6a425e99392ba0aaac6d7a08aad4c1: Status 404 returned error can't find the container with id 3ac6dba1f7d8f5e6dc6b72e614afb9042e6a425e99392ba0aaac6d7a08aad4c1 Nov 01 20:40:26.782970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:26.782939 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" event={"ID":"70e0106a-3e12-4a6d-a721-2a11082f8e7b","Type":"ContainerStarted","Data":"3ac6dba1f7d8f5e6dc6b72e614afb9042e6a425e99392ba0aaac6d7a08aad4c1"} Nov 01 20:40:29.802468 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:29.802430 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" event={"ID":"70e0106a-3e12-4a6d-a721-2a11082f8e7b","Type":"ContainerStarted","Data":"a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007"} Nov 01 20:40:29.802991 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:29.802563 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:29.835691 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:29.835661 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf"] Nov 01 20:40:29.835904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:29.835870 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" containerID="cri-o://4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97" gracePeriod=30 Nov 01 20:40:29.918069 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:29.918018 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" podStartSLOduration=1.830970766 podStartE2EDuration="4.918005593s" totalImagesPullingTime="3.087034827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:29.916805206 +0000 UTC m=+2230.861643074" watchObservedRunningTime="2026-11-01 20:40:29.918005593 +0000 UTC m=+2230.862843480" Nov 01 20:40:30.071753 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:30.071577 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:40:35.083818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.083794 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:40:35.218866 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.218808 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.218970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.218869 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.218970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.218923 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219051 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.218970 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219051 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219005 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219051 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219043 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219069 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219121 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219205 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219149 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token\") pod \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\" (UID: \"a15b1f8b-b1c4-48e9-bd69-96096a751ac6\") " Nov 01 20:40:35.219363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219221 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.219363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219238 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.219363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219271 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.219540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219463 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.219540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219483 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.219540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219498 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.219540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219513 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.219540 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.219519 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:35.221134 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.221111 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "kube-api-access-vlhfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:35.221229 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.221183 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:35.221363 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.221337 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:40:35.221432 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.221390 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token" pod "a15b1f8b-b1c4-48e9-bd69-96096a751ac6" (UID: "a15b1f8b-b1c4-48e9-bd69-96096a751ac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:35.320348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.320317 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhfc\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-kube-api-access-vlhfc\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.320348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.320345 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.320348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.320355 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.320559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.320364 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.320559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.320374 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.320559 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.320382 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a15b1f8b-b1c4-48e9-bd69-96096a751ac6-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:35.827965 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.827932 5444 generic.go:356] "Generic (PLEG): container finished" podID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerID="4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97" exitCode=0 Nov 01 20:40:35.828190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.827977 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" Nov 01 20:40:35.828190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.828015 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" event={"ID":"a15b1f8b-b1c4-48e9-bd69-96096a751ac6","Type":"ContainerDied","Data":"4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97"} Nov 01 20:40:35.828190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.828078 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf" event={"ID":"a15b1f8b-b1c4-48e9-bd69-96096a751ac6","Type":"ContainerDied","Data":"a7c69132547f922e4c9f85890d6fc6de589ae69b89cb2fcf3f42187ce40526ef"} Nov 01 20:40:35.828190 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.828109 5444 scope.go:122] "RemoveContainer" containerID="4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97" Nov 01 20:40:35.837269 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.837253 5444 scope.go:122] "RemoveContainer" containerID="4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97" Nov 01 20:40:35.837526 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:35.837509 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97\": container with ID starting with 4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97 not found: ID does not exist" containerID="4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97" Nov 01 20:40:35.837576 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:35.837534 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97"} err="failed to get container status \"4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97\": rpc error: code = NotFound desc = could not find container \"4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97\": container with ID starting with 4d4ac75a670c177422568f8f2c2a8c43de1f389e1e443cb9b07bbbeef22b6d97 not found: ID does not exist" Nov 01 20:40:36.039708 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:36.039682 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf"] Nov 01 20:40:36.042521 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:36.042500 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1b1134-openshift-default-7bbbbc6687-mchgf"] Nov 01 20:40:37.157784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.157742 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" path="/var/lib/kubelet/pods/a15b1f8b-b1c4-48e9-bd69-96096a751ac6/volumes" Nov 01 20:40:37.895687 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.895658 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq"] Nov 01 20:40:37.898571 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.898541 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" podUID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:40:37.944756 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.944730 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr"] Nov 01 20:40:37.945056 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.945044 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" Nov 01 20:40:37.945110 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.945057 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" Nov 01 20:40:37.945153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.945135 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15b1f8b-b1c4-48e9-bd69-96096a751ac6" containerName="istio-proxy" Nov 01 20:40:37.949446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.949428 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:37.979450 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:37.979429 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr"] Nov 01 20:40:38.045545 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.045514 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.045725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.045573 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdvz\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.045725 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.045658 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.045859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.045779 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.146659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.146586 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdvz\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.146659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.146614 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.146862 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.146681 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.146862 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.146705 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.147329 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.147307 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.147413 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.147332 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.148945 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.148926 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.202797 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.202774 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdvz\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.261288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.261262 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.628952 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.628814 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr"] Nov 01 20:40:38.631758 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:40:38.631727 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566f6bc6_852d_4a82_beaa_fada533e94d1.slice/crio-0000b63e38733b590ac890890c1142816732ed130597683be72712fa625fe299 WatchSource:0}: Error finding container 0000b63e38733b590ac890890c1142816732ed130597683be72712fa625fe299: Status 404 returned error can't find the container with id 0000b63e38733b590ac890890c1142816732ed130597683be72712fa625fe299 Nov 01 20:40:38.844338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.844289 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" event={"ID":"566f6bc6-852d-4a82-beaa-fada533e94d1","Type":"ContainerStarted","Data":"c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b"} Nov 01 20:40:38.844493 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.844447 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:38.844493 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.844466 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" event={"ID":"566f6bc6-852d-4a82-beaa-fada533e94d1","Type":"ContainerStarted","Data":"0000b63e38733b590ac890890c1142816732ed130597683be72712fa625fe299"} Nov 01 20:40:38.956048 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:38.955975 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" podStartSLOduration=1.955963001 podStartE2EDuration="1.955963001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:38.955824265 +0000 UTC m=+2239.900662136" watchObservedRunningTime="2026-11-01 20:40:38.955963001 +0000 UTC m=+2239.900800869" Nov 01 20:40:42.896478 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:42.896437 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" podUID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" containerName="payload-processing" containerID="cri-o://a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007" gracePeriod=25 Nov 01 20:40:43.156870 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.156808 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:43.290031 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290004 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume\") pod \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " Nov 01 20:40:43.290152 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290076 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume\") pod \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " Nov 01 20:40:43.290152 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290135 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq\") pod \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " Nov 01 20:40:43.290225 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290166 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials\") pod \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\" (UID: \"70e0106a-3e12-4a6d-a721-2a11082f8e7b\") " Nov 01 20:40:43.290828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290418 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume" pod "70e0106a-3e12-4a6d-a721-2a11082f8e7b" (UID: "70e0106a-3e12-4a6d-a721-2a11082f8e7b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:43.290828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290447 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume" pod "70e0106a-3e12-4a6d-a721-2a11082f8e7b" (UID: "70e0106a-3e12-4a6d-a721-2a11082f8e7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:43.290828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290525 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:43.290828 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.290545 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/70e0106a-3e12-4a6d-a721-2a11082f8e7b-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:43.292419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.292389 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq" pod "70e0106a-3e12-4a6d-a721-2a11082f8e7b" (UID: "70e0106a-3e12-4a6d-a721-2a11082f8e7b"). InnerVolumeSpecName "kube-api-access-tzkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:43.292419 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.292404 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials" pod "70e0106a-3e12-4a6d-a721-2a11082f8e7b" (UID: "70e0106a-3e12-4a6d-a721-2a11082f8e7b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:43.391235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.391213 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzkpq\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-kube-api-access-tzkpq\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:43.391235 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.391233 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/70e0106a-3e12-4a6d-a721-2a11082f8e7b-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:43.869070 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.869034 5444 generic.go:356] "Generic (PLEG): container finished" podID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" containerID="a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007" exitCode=0 Nov 01 20:40:43.869241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.869088 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" Nov 01 20:40:43.869241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.869096 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" event={"ID":"70e0106a-3e12-4a6d-a721-2a11082f8e7b","Type":"ContainerDied","Data":"a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007"} Nov 01 20:40:43.869241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.869140 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq" event={"ID":"70e0106a-3e12-4a6d-a721-2a11082f8e7b","Type":"ContainerDied","Data":"3ac6dba1f7d8f5e6dc6b72e614afb9042e6a425e99392ba0aaac6d7a08aad4c1"} Nov 01 20:40:43.869241 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.869162 5444 scope.go:122] "RemoveContainer" containerID="a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007" Nov 01 20:40:43.878529 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.878511 5444 scope.go:122] "RemoveContainer" containerID="a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007" Nov 01 20:40:43.878794 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:43.878777 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007\": container with ID starting with a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007 not found: ID does not exist" containerID="a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007" Nov 01 20:40:43.878847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:43.878803 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007"} err="failed to get container status \"a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007\": rpc error: code = NotFound desc = could not find container \"a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007\": container with ID starting with a4a4cd30494d1873477590029adcb28a4022cc690490fb079df553595d0b2007 not found: ID does not exist" Nov 01 20:40:44.030700 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:44.030672 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq"] Nov 01 20:40:44.080140 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:44.080118 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-586bcc4lq"] Nov 01 20:40:45.157074 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:45.157040 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" path="/var/lib/kubelet/pods/70e0106a-3e12-4a6d-a721-2a11082f8e7b/volumes" Nov 01 20:40:48.502457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.502428 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr"] Nov 01 20:40:48.505022 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.504992 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" podUID="566f6bc6-852d-4a82-beaa-fada533e94d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:40:48.551720 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.551689 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7"] Nov 01 20:40:48.552021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.552007 5444 cpu_manager.go:414] "containerMap: removing container" podUID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" containerName="payload-processing" Nov 01 20:40:48.552021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.552020 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" containerName="payload-processing" Nov 01 20:40:48.552150 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.552088 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e0106a-3e12-4a6d-a721-2a11082f8e7b" containerName="payload-processing" Nov 01 20:40:48.556610 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.556587 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.607729 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.607709 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-164004586dcd\"" Nov 01 20:40:48.615818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.615795 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7"] Nov 01 20:40:48.648722 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.648690 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.648845 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.648742 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvf5s\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.648928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.648837 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.648928 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.648890 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.749418 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.749380 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.749592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.749441 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.749592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.749486 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvf5s\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.749592 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.749549 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.750258 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.750228 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.750351 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.750286 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.752060 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.752042 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.805405 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.805374 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvf5s\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:48.868976 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:48.868948 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:49.236044 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:49.235974 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7"] Nov 01 20:40:49.239221 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:40:49.239194 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a444b91_c245_4a6c_9ab6_95d2610358de.slice/crio-17be04e9cf096da2542a4a1797b5aece78131a2cf0e6b045cb0c2ef8e0912264 WatchSource:0}: Error finding container 17be04e9cf096da2542a4a1797b5aece78131a2cf0e6b045cb0c2ef8e0912264: Status 404 returned error can't find the container with id 17be04e9cf096da2542a4a1797b5aece78131a2cf0e6b045cb0c2ef8e0912264 Nov 01 20:40:49.900350 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:49.900279 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" event={"ID":"3a444b91-c245-4a6c-9ab6-95d2610358de","Type":"ContainerStarted","Data":"6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951"} Nov 01 20:40:49.900350 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:49.900336 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" event={"ID":"3a444b91-c245-4a6c-9ab6-95d2610358de","Type":"ContainerStarted","Data":"17be04e9cf096da2542a4a1797b5aece78131a2cf0e6b045cb0c2ef8e0912264"} Nov 01 20:40:49.900738 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:49.900364 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:40:50.013903 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:50.013856 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" podStartSLOduration=2.013843132 podStartE2EDuration="2.013843132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:50.012061423 +0000 UTC m=+2250.956899290" watchObservedRunningTime="2026-11-01 20:40:50.013843132 +0000 UTC m=+2250.958681000" Nov 01 20:40:53.503021 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.502979 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" podUID="566f6bc6-852d-4a82-beaa-fada533e94d1" containerName="payload-processing" containerID="cri-o://c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b" gracePeriod=25 Nov 01 20:40:53.750500 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.750474 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:53.793843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.793820 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume\") pod \"566f6bc6-852d-4a82-beaa-fada533e94d1\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " Nov 01 20:40:53.793981 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.793848 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume\") pod \"566f6bc6-852d-4a82-beaa-fada533e94d1\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " Nov 01 20:40:53.793981 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.793897 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz\") pod \"566f6bc6-852d-4a82-beaa-fada533e94d1\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " Nov 01 20:40:53.794101 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.793993 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials\") pod \"566f6bc6-852d-4a82-beaa-fada533e94d1\" (UID: \"566f6bc6-852d-4a82-beaa-fada533e94d1\") " Nov 01 20:40:53.794226 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.794202 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume" pod "566f6bc6-852d-4a82-beaa-fada533e94d1" (UID: "566f6bc6-852d-4a82-beaa-fada533e94d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:53.794293 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.794252 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume" pod "566f6bc6-852d-4a82-beaa-fada533e94d1" (UID: "566f6bc6-852d-4a82-beaa-fada533e94d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:40:53.795894 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.795874 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials" pod "566f6bc6-852d-4a82-beaa-fada533e94d1" (UID: "566f6bc6-852d-4a82-beaa-fada533e94d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:53.795993 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.795974 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz" pod "566f6bc6-852d-4a82-beaa-fada533e94d1" (UID: "566f6bc6-852d-4a82-beaa-fada533e94d1"). InnerVolumeSpecName "kube-api-access-2jdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:53.895582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.895555 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdvz\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-kube-api-access-2jdvz\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:53.895582 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.895581 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/566f6bc6-852d-4a82-beaa-fada533e94d1-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:53.895732 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.895597 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:53.895732 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.895610 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566f6bc6-852d-4a82-beaa-fada533e94d1-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:40:53.919106 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.919085 5444 generic.go:356] "Generic (PLEG): container finished" podID="566f6bc6-852d-4a82-beaa-fada533e94d1" containerID="c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b" exitCode=0 Nov 01 20:40:53.919210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.919126 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" Nov 01 20:40:53.919210 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.919164 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" event={"ID":"566f6bc6-852d-4a82-beaa-fada533e94d1","Type":"ContainerDied","Data":"c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b"} Nov 01 20:40:53.919284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.919215 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr" event={"ID":"566f6bc6-852d-4a82-beaa-fada533e94d1","Type":"ContainerDied","Data":"0000b63e38733b590ac890890c1142816732ed130597683be72712fa625fe299"} Nov 01 20:40:53.919284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.919238 5444 scope.go:122] "RemoveContainer" containerID="c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b" Nov 01 20:40:53.927970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.927956 5444 scope.go:122] "RemoveContainer" containerID="c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b" Nov 01 20:40:53.928212 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:40:53.928197 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b\": container with ID starting with c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b not found: ID does not exist" containerID="c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b" Nov 01 20:40:53.928258 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:53.928220 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b"} err="failed to get container status \"c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b\": rpc error: code = NotFound desc = could not find container \"c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b\": container with ID starting with c1f3951cb19519a13477c4b92d4b5d4783d86203e18fcf0d96dbb72b9afee31b not found: ID does not exist" Nov 01 20:40:54.080795 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:54.080735 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr"] Nov 01 20:40:54.084084 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:54.084063 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-6b5dhgwnr"] Nov 01 20:40:55.156424 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:40:55.156391 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566f6bc6-852d-4a82-beaa-fada533e94d1" path="/var/lib/kubelet/pods/566f6bc6-852d-4a82-beaa-fada533e94d1/volumes" Nov 01 20:41:00.033471 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:41:00.033433 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:41:00.910185 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:00.910155 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:41:20.168707 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.168650 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp"] Nov 01 20:41:20.169048 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.168980 5444 cpu_manager.go:414] "containerMap: removing container" podUID="566f6bc6-852d-4a82-beaa-fada533e94d1" containerName="payload-processing" Nov 01 20:41:20.169048 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.168991 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566f6bc6-852d-4a82-beaa-fada533e94d1" containerName="payload-processing" Nov 01 20:41:20.169122 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.169068 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="566f6bc6-852d-4a82-beaa-fada533e94d1" containerName="payload-processing" Nov 01 20:41:20.172914 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.172898 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.224818 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.224790 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6a675a-openshift-default-dockercfg-85chn\"" Nov 01 20:41:20.275542 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.275517 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp"] Nov 01 20:41:20.303874 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.303848 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.303992 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.303892 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.303992 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.303914 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.304125 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.304017 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.304125 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.304071 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.304238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.304147 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.304238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.304184 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnpj\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.304238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.304207 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.304238 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.304230 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405196 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405169 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405210 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405243 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnpj\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405272 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405314 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405297 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405325 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405381 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405418 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405455 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405787 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405709 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405846 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405805 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405848 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.405904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.405876 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.406095 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.406077 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.407566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.407544 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.407927 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.407904 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.461572 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.461520 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.461683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.461667 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnpj\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj\") pod \"e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.484948 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.484928 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:20.960849 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.960513 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp"] Nov 01 20:41:20.963257 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:41:20.963231 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc65e78_2499_4005_85a1_a21797d45aa3.slice/crio-f74f34593010b9e5f358e06255b7a3bafcd92968fdcf0988b474a36f86844e14 WatchSource:0}: Error finding container f74f34593010b9e5f358e06255b7a3bafcd92968fdcf0988b474a36f86844e14: Status 404 returned error can't find the container with id f74f34593010b9e5f358e06255b7a3bafcd92968fdcf0988b474a36f86844e14 Nov 01 20:41:20.965446 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.965416 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:20.965521 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.965478 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:20.965521 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:20.965510 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:21.046764 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:21.046736 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" event={"ID":"3fc65e78-2499-4005-85a1-a21797d45aa3","Type":"ContainerStarted","Data":"43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9"} Nov 01 20:41:21.046905 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:21.046797 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" event={"ID":"3fc65e78-2499-4005-85a1-a21797d45aa3","Type":"ContainerStarted","Data":"f74f34593010b9e5f358e06255b7a3bafcd92968fdcf0988b474a36f86844e14"} Nov 01 20:41:21.164709 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:21.164663 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podStartSLOduration=1.164628851 podStartE2EDuration="1.164628851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:21.163710342 +0000 UTC m=+2282.108548210" watchObservedRunningTime="2026-11-01 20:41:21.164628851 +0000 UTC m=+2282.109466716" Nov 01 20:41:21.485899 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:21.485863 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:22.487230 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:22.487195 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:41:22.487601 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:22.487258 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:41:23.485306 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:23.485270 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:41:23.485475 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:23.485338 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:41:24.489186 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:24.489150 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:41:24.489831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:24.489784 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:41:25.485560 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:25.485529 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:41:25.485765 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:25.485733 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:41:26.486347 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:26.486307 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:41:26.486772 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:26.486369 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:41:27.489509 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:27.489478 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:27.489964 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:27.489869 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:27.490567 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:41:27.490549 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:41:30.035151 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:41:30.035114 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:42:00.034933 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:42:00.034899 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:42:30.035189 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:42:30.035153 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:43:00.034363 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:00.034320 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:43:30.034074 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:30.034030 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:43:33.279043 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:33.279000 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:43:33.279043 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:33.279047 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:43:33.279612 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:33.279123 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:43:33.77910544 +0000 UTC m=+2414.723943299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:43:33.784654 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:33.784609 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:43:33.784835 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:33.784663 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:43:33.784835 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:33.784727 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:43:34.784713342 +0000 UTC m=+2415.729551202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:43:34.792616 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:34.792580 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:43:34.792616 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:34.792620 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:43:34.793094 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:34.792688 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:43:36.79267248 +0000 UTC m=+2417.737510327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:43:36.810913 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:36.810875 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:43:36.810913 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:36.810913 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:43:36.811340 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:36.810965 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:43:40.810952269 +0000 UTC m=+2421.755790128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:43:40.848318 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:40.848264 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:43:40.848318 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:40.848318 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:43:40.848792 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:40.848386 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:43:48.848370385 +0000 UTC m=+2429.793208232 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:43:48.930967 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:48.930933 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:43:48.930967 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:48.930968 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:43:48.931381 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:43:48.931029 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:44:04.931012203 +0000 UTC m=+2445.875850069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:44:00.034796 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:00.034706 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:44:04.972781 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:04.972747 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:44:04.972781 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:04.972786 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: secret "agc-cred-164004586dcd" not found Nov 01 20:44:04.973401 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:04.972881 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:44:36.972860752 +0000 UTC m=+2477.917698609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : secret "agc-cred-164004586dcd" not found Nov 01 20:44:30.034112 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:30.034078 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:44:37.058496 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:37.058463 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-164004586dcd: secret "agc-cred-164004586dcd" not found Nov 01 20:44:37.058496 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:37.058496 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-cd18c1528ab8: secret "agc-cred-cd18c1528ab8" not found Nov 01 20:44:37.058496 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:37.058504 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6a675a/agc-cred-e385637602d5: secret "agc-cred-e385637602d5" not found Nov 01 20:44:37.059063 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:37.058515 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7: [secret "agc-cred-164004586dcd" not found, secret "agc-cred-cd18c1528ab8" not found, secret "agc-cred-e385637602d5" not found] Nov 01 20:44:37.059063 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:37.058580 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials podName:3a444b91-c245-4a6c-9ab6-95d2610358de nodeName:}" failed. No retries permitted until 2026-11-01 20:45:41.05856606 +0000 UTC m=+2542.003403906 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de") : [secret "agc-cred-164004586dcd" not found, secret "agc-cred-cd18c1528ab8" not found, secret "agc-cred-e385637602d5" not found] Nov 01 20:44:39.389514 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:39.389475 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4"] Nov 01 20:44:39.389964 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:39.389689 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" containerID="cri-o://92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02" gracePeriod=30 Nov 01 20:44:44.633480 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.633456 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:44:44.737628 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737541 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.737628 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737600 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.737847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737656 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.737847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737697 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.737847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737739 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.737847 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737803 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.738059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737852 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.738059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737952 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:44:44.738059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.737980 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:44:44.738059 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738009 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.738276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738092 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data\") pod \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\" (UID: \"d7edbe4e-7819-4702-a5e4-77ca37e0004a\") " Nov 01 20:44:44.738276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738105 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:44.738276 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738106 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:44:44.738425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738411 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:44:44.738485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738436 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.738485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738456 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.738485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738469 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.738485 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.738482 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.740055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.740021 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "kube-api-access-gvmrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:44.740055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.740039 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:44.740055 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.740051 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:44:44.740232 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.740144 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo" pod "d7edbe4e-7819-4702-a5e4-77ca37e0004a" (UID: "d7edbe4e-7819-4702-a5e4-77ca37e0004a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:44:44.839669 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.839615 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.839669 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.839664 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmrr\" (UniqueName: \"kubernetes.io/projected/d7edbe4e-7819-4702-a5e4-77ca37e0004a-kube-api-access-gvmrr\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.839669 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.839676 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.839907 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.839685 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.839907 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.839695 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7edbe4e-7819-4702-a5e4-77ca37e0004a-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:44.936810 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.936775 5444 generic.go:356] "Generic (PLEG): container finished" podID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerID="92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02" exitCode=0 Nov 01 20:44:44.936960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.936833 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" Nov 01 20:44:44.936960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.936831 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" event={"ID":"d7edbe4e-7819-4702-a5e4-77ca37e0004a","Type":"ContainerDied","Data":"92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02"} Nov 01 20:44:44.936960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.936885 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4" event={"ID":"d7edbe4e-7819-4702-a5e4-77ca37e0004a","Type":"ContainerDied","Data":"392622527f0d66c707a60a36aeb263c6b2b0d1d5d3f4123f0f13209663c3d6aa"} Nov 01 20:44:44.936960 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.936914 5444 scope.go:122] "RemoveContainer" containerID="92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02" Nov 01 20:44:44.946062 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.946041 5444 scope.go:122] "RemoveContainer" containerID="92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02" Nov 01 20:44:44.946315 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:44.946294 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02\": container with ID starting with 92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02 not found: ID does not exist" containerID="92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02" Nov 01 20:44:44.946366 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:44.946326 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02"} err="failed to get container status \"92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02\": rpc error: code = NotFound desc = could not find container \"92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02\": container with ID starting with 92fac0ee229f5d699a1235dbbcbc53301d0c2e2343ec59c41a9726e1bd2e2a02 not found: ID does not exist" Nov 01 20:44:45.100843 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:45.100807 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4"] Nov 01 20:44:45.108613 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:45.108584 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-df41b8-openshift-default-cb95f544f-8tdl4"] Nov 01 20:44:45.157671 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:45.157630 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" path="/var/lib/kubelet/pods/d7edbe4e-7819-4702-a5e4-77ca37e0004a/volumes" Nov 01 20:44:45.284070 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:45.284039 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7"] Nov 01 20:44:50.285290 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.285218 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" podUID="3a444b91-c245-4a6c-9ab6-95d2610358de" containerName="payload-processing" containerID="cri-o://6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951" gracePeriod=25 Nov 01 20:44:50.536367 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.536312 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:44:50.695649 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.695617 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume\") pod \"3a444b91-c245-4a6c-9ab6-95d2610358de\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " Nov 01 20:44:50.695831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.695700 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume\") pod \"3a444b91-c245-4a6c-9ab6-95d2610358de\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " Nov 01 20:44:50.695831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.695749 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s\") pod \"3a444b91-c245-4a6c-9ab6-95d2610358de\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " Nov 01 20:44:50.695831 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.695816 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials\") pod \"3a444b91-c245-4a6c-9ab6-95d2610358de\" (UID: \"3a444b91-c245-4a6c-9ab6-95d2610358de\") " Nov 01 20:44:50.696023 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.695961 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume" pod "3a444b91-c245-4a6c-9ab6-95d2610358de" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:50.696151 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.696128 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume" pod "3a444b91-c245-4a6c-9ab6-95d2610358de" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:44:50.697864 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.697839 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials" pod "3a444b91-c245-4a6c-9ab6-95d2610358de" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:50.697956 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.697942 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s" pod "3a444b91-c245-4a6c-9ab6-95d2610358de" (UID: "3a444b91-c245-4a6c-9ab6-95d2610358de"). InnerVolumeSpecName "kube-api-access-nvf5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:44:50.797224 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.797139 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:50.797224 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.797171 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:50.797224 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.797182 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a444b91-c245-4a6c-9ab6-95d2610358de-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:50.797224 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.797191 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvf5s\" (UniqueName: \"kubernetes.io/projected/3a444b91-c245-4a6c-9ab6-95d2610358de-kube-api-access-nvf5s\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:44:50.992373 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.992337 5444 generic.go:356] "Generic (PLEG): container finished" podID="3a444b91-c245-4a6c-9ab6-95d2610358de" containerID="6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951" exitCode=0 Nov 01 20:44:50.992546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.992394 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" Nov 01 20:44:50.992546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.992418 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" event={"ID":"3a444b91-c245-4a6c-9ab6-95d2610358de","Type":"ContainerDied","Data":"6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951"} Nov 01 20:44:50.992546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.992472 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7" event={"ID":"3a444b91-c245-4a6c-9ab6-95d2610358de","Type":"ContainerDied","Data":"17be04e9cf096da2542a4a1797b5aece78131a2cf0e6b045cb0c2ef8e0912264"} Nov 01 20:44:50.992546 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:50.992495 5444 scope.go:122] "RemoveContainer" containerID="6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951" Nov 01 20:44:51.001239 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:51.001218 5444 scope.go:122] "RemoveContainer" containerID="6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951" Nov 01 20:44:51.001536 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:44:51.001518 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951\": container with ID starting with 6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951 not found: ID does not exist" containerID="6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951" Nov 01 20:44:51.001597 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:51.001544 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951"} err="failed to get container status \"6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951\": rpc error: code = NotFound desc = could not find container \"6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951\": container with ID starting with 6e397edb9f35f90fe8a06fe3aa5e20fccc6a95ee28c03fea4450edee55375951 not found: ID does not exist" Nov 01 20:44:51.157065 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:51.156989 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7"] Nov 01 20:44:51.202239 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:51.202212 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-54d8jd5p7"] Nov 01 20:44:53.157153 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:44:53.157120 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a444b91-c245-4a6c-9ab6-95d2610358de" path="/var/lib/kubelet/pods/3a444b91-c245-4a6c-9ab6-95d2610358de/volumes" Nov 01 20:45:00.034028 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:45:00.033990 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:45:16.144626 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.144593 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj"] Nov 01 20:45:16.145160 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.144833 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" podUID="6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" containerName="maas-api" containerID="cri-o://1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6" gracePeriod=30 Nov 01 20:45:16.426991 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.426967 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:45:16.489433 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.489408 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\") pod \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " Nov 01 20:45:16.489552 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.489445 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca\") pod \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " Nov 01 20:45:16.489886 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.489864 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca" pod "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" (UID: "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:16.491468 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.491449 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls" pod "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" (UID: "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:16.590589 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.590558 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs\") pod \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " Nov 01 20:45:16.590784 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.590601 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g\" (UniqueName: \"kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g\") pod \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\" (UID: \"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555\") " Nov 01 20:45:16.590859 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.590846 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:16.590898 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.590862 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:16.592629 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.592601 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs" pod "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" (UID: "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:16.592756 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.592629 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g" pod "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" (UID: "6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555"). InnerVolumeSpecName "kube-api-access-s5n2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:16.692133 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.692056 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:16.692133 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:16.692086 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5n2g\" (UniqueName: \"kubernetes.io/projected/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555-kube-api-access-s5n2g\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:17.157685 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.157622 5444 generic.go:356] "Generic (PLEG): container finished" podID="6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" containerID="1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6" exitCode=0 Nov 01 20:45:17.158146 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.157714 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" Nov 01 20:45:17.158736 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.158712 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" event={"ID":"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555","Type":"ContainerDied","Data":"1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6"} Nov 01 20:45:17.158885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.158809 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj" event={"ID":"6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555","Type":"ContainerDied","Data":"80be74b15a3b34fa2d176f31048f1948ec1e4921566a1103a92d3c6decb2e10a"} Nov 01 20:45:17.158885 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.158846 5444 scope.go:122] "RemoveContainer" containerID="1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6" Nov 01 20:45:17.169942 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.169924 5444 scope.go:122] "RemoveContainer" containerID="1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6" Nov 01 20:45:17.170181 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:45:17.170163 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6\": container with ID starting with 1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6 not found: ID does not exist" containerID="1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6" Nov 01 20:45:17.170242 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.170191 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6"} err="failed to get container status \"1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6\": rpc error: code = NotFound desc = could not find container \"1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6\": container with ID starting with 1d626b914f52dbdf40f2da5d9d388b67a383a83b63aecc9e9cff7e2919fc58d6 not found: ID does not exist" Nov 01 20:45:17.319463 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.319431 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj"] Nov 01 20:45:17.368593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:17.368561 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-df41b8-cbf765dd5-rl6bj"] Nov 01 20:45:19.157115 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:19.157082 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" path="/var/lib/kubelet/pods/6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555/volumes" Nov 01 20:45:30.033999 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:45:30.033960 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:45:46.205072 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.205041 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5"] Nov 01 20:45:46.205530 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.205259 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" podUID="32ac1051-c4df-4936-badc-06620530e792" containerName="maas-api" containerID="cri-o://0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276" gracePeriod=30 Nov 01 20:45:46.461408 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.461349 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:45:46.599025 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.598976 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca\") pod \"32ac1051-c4df-4936-badc-06620530e792\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " Nov 01 20:45:46.599315 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.599296 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4\" (UniqueName: \"kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4\") pod \"32ac1051-c4df-4936-badc-06620530e792\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " Nov 01 20:45:46.599453 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.599439 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\") pod \"32ac1051-c4df-4936-badc-06620530e792\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " Nov 01 20:45:46.603889 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.602008 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca" pod "32ac1051-c4df-4936-badc-06620530e792" (UID: "32ac1051-c4df-4936-badc-06620530e792"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:46.604031 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.603937 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs\") pod \"32ac1051-c4df-4936-badc-06620530e792\" (UID: \"32ac1051-c4df-4936-badc-06620530e792\") " Nov 01 20:45:46.604284 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.604261 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ac1051-c4df-4936-badc-06620530e792-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:46.605280 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.605248 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4" pod "32ac1051-c4df-4936-badc-06620530e792" (UID: "32ac1051-c4df-4936-badc-06620530e792"). InnerVolumeSpecName "kube-api-access-clrn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:46.605425 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.605251 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls" pod "32ac1051-c4df-4936-badc-06620530e792" (UID: "32ac1051-c4df-4936-badc-06620530e792"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:46.606498 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.606471 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs" pod "32ac1051-c4df-4936-badc-06620530e792" (UID: "32ac1051-c4df-4936-badc-06620530e792"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:46.705066 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.705028 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:46.705066 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.705068 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ac1051-c4df-4936-badc-06620530e792-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:46.705288 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:46.705081 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrn4\" (UniqueName: \"kubernetes.io/projected/32ac1051-c4df-4936-badc-06620530e792-kube-api-access-clrn4\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:45:47.292857 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.292824 5444 generic.go:356] "Generic (PLEG): container finished" podID="32ac1051-c4df-4936-badc-06620530e792" containerID="0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276" exitCode=0 Nov 01 20:45:47.293355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.292865 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" Nov 01 20:45:47.293355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.292873 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" event={"ID":"32ac1051-c4df-4936-badc-06620530e792","Type":"ContainerDied","Data":"0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276"} Nov 01 20:45:47.293355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.292924 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5" event={"ID":"32ac1051-c4df-4936-badc-06620530e792","Type":"ContainerDied","Data":"b031adfc4134527e020d1c8be6c439a4d505cd4726c5e91defe557f7609127ad"} Nov 01 20:45:47.293355 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.292954 5444 scope.go:122] "RemoveContainer" containerID="0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276" Nov 01 20:45:47.301336 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.301317 5444 scope.go:122] "RemoveContainer" containerID="0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276" Nov 01 20:45:47.301587 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:45:47.301568 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276\": container with ID starting with 0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276 not found: ID does not exist" containerID="0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276" Nov 01 20:45:47.301659 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.301598 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276"} err="failed to get container status \"0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276\": rpc error: code = NotFound desc = could not find container \"0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276\": container with ID starting with 0eb8d8a8f4d2e364a46222c97098c66b4972239f77df7fac9477f9b9d020f276 not found: ID does not exist" Nov 01 20:45:47.455932 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.455840 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5"] Nov 01 20:45:47.503783 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:47.503754 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc24d-6b69cdcd49-fl9v5"] Nov 01 20:45:49.156970 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:45:49.156936 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ac1051-c4df-4936-badc-06620530e792" path="/var/lib/kubelet/pods/32ac1051-c4df-4936-badc-06620530e792/volumes" Nov 01 20:46:00.036676 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:46:00.036624 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:46:30.034926 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:46:30.034888 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:46:37.483748 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:37.483715 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp"] Nov 01 20:46:37.484199 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:37.483957 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" containerID="cri-o://43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9" gracePeriod=30 Nov 01 20:46:42.499313 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.499275 5444 patch_prober.go:29] interesting pod/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": read tcp 10.133.0.2:48954->10.133.0.81:15021: read: connection reset by peer" start-of-body= Nov 01 20:46:42.499844 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.499340 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": read tcp 10.133.0.2:48954->10.133.0.81:15021: read: connection reset by peer" Nov 01 20:46:42.731471 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.731447 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:46:42.788013 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.787980 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.788207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.788054 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.788207 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.788097 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.788334 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.788245 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:42.788387 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.788348 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:42.788459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.788440 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.788505 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.788464 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.790100 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.790081 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:42.889587 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.889556 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.889791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.889604 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.889791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.889661 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.889791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.889747 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.889791 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.889779 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.890019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.889805 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj\") pod \"3fc65e78-2499-4005-85a1-a21797d45aa3\" (UID: \"3fc65e78-2499-4005-85a1-a21797d45aa3\") " Nov 01 20:46:42.890078 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.890029 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:42.890078 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.890048 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:42.890192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.890115 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:46:42.890192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.890135 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.890192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.890153 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.890192 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.890167 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.891814 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.891784 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:46:42.891904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.891857 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:46:42.891904 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.891877 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj" pod "3fc65e78-2499-4005-85a1-a21797d45aa3" (UID: "3fc65e78-2499-4005-85a1-a21797d45aa3"). InnerVolumeSpecName "kube-api-access-5cnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:46:42.991482 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.991451 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.991482 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.991478 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc65e78-2499-4005-85a1-a21797d45aa3-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.991724 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.991492 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnpj\" (UniqueName: \"kubernetes.io/projected/3fc65e78-2499-4005-85a1-a21797d45aa3-kube-api-access-5cnpj\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:42.991724 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:42.991503 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc65e78-2499-4005-85a1-a21797d45aa3-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:46:43.536606 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.536574 5444 generic.go:356] "Generic (PLEG): container finished" podID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerID="43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9" exitCode=0 Nov 01 20:46:43.537019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.536645 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" Nov 01 20:46:43.537019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.536633 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" event={"ID":"3fc65e78-2499-4005-85a1-a21797d45aa3","Type":"ContainerDied","Data":"43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9"} Nov 01 20:46:43.537019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.536698 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp" event={"ID":"3fc65e78-2499-4005-85a1-a21797d45aa3","Type":"ContainerDied","Data":"f74f34593010b9e5f358e06255b7a3bafcd92968fdcf0988b474a36f86844e14"} Nov 01 20:46:43.537019 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.536720 5444 scope.go:122] "RemoveContainer" containerID="43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9" Nov 01 20:46:43.544783 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.544765 5444 scope.go:122] "RemoveContainer" containerID="43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9" Nov 01 20:46:43.545033 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:46:43.545015 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9\": container with ID starting with 43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9 not found: ID does not exist" containerID="43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9" Nov 01 20:46:43.545075 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.545042 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9"} err="failed to get container status \"43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9\": rpc error: code = NotFound desc = could not find container \"43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9\": container with ID starting with 43986332e4b7d21f663ed863342431a7762217b1f0a89846bcb741be8ef6bcc9 not found: ID does not exist" Nov 01 20:46:43.748554 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.748519 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp"] Nov 01 20:46:43.797972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:43.797906 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6a675a-openshift-default-59cff7cb5-srzvp"] Nov 01 20:46:45.156712 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:46:45.156678 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" path="/var/lib/kubelet/pods/3fc65e78-2499-4005-85a1-a21797d45aa3/volumes" Nov 01 20:47:00.034364 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:47:00.034282 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:47:30.034324 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:47:30.034286 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:48:00.034709 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:48:00.034665 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:48:30.034108 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:48:30.034064 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:48:33.750457 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750418 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54"] Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750766 5444 cpu_manager.go:414] "containerMap: removing container" podUID="32ac1051-c4df-4936-badc-06620530e792" containerName="maas-api" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750781 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ac1051-c4df-4936-badc-06620530e792" containerName="maas-api" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750800 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750806 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750825 5444 cpu_manager.go:414] "containerMap: removing container" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750830 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750842 5444 cpu_manager.go:414] "containerMap: removing container" podUID="6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" containerName="maas-api" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750848 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" containerName="maas-api" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750855 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3a444b91-c245-4a6c-9ab6-95d2610358de" containerName="payload-processing" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750860 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a444b91-c245-4a6c-9ab6-95d2610358de" containerName="payload-processing" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750919 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecdc2db-0bfa-4b3f-8d4d-af9cae29f555" containerName="maas-api" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750925 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a444b91-c245-4a6c-9ab6-95d2610358de" containerName="payload-processing" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750934 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ac1051-c4df-4936-badc-06620530e792" containerName="maas-api" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750941 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc65e78-2499-4005-85a1-a21797d45aa3" containerName="istio-proxy" Nov 01 20:48:33.750954 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.750949 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7edbe4e-7819-4702-a5e4-77ca37e0004a" containerName="istio-proxy" Nov 01 20:48:33.753919 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.753899 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.828439 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.828407 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.828580 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.828457 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.828580 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.828518 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qbr\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.828580 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.828538 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.855593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855565 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cfl-224e35-provider-cred\"" Nov 01 20:48:33.855740 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855707 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dhzm4\"" Nov 01 20:48:33.855740 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855734 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 20:48:33.855864 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855748 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-1c1bf2169d38\"" Nov 01 20:48:33.855906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855859 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 20:48:33.855906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855875 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 20:48:33.855906 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.855885 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 20:48:33.856951 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.856933 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54"] Nov 01 20:48:33.929420 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.929398 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qbr\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.929538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.929428 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.929538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.929472 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.929538 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.929504 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.930180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.930157 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.930251 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.930164 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:33.931981 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:33.931958 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:34.003107 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:34.003057 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qbr\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:34.067274 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:34.067253 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:34.461525 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:34.461489 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54"] Nov 01 20:48:34.465426 ip-10-0-138-177 kubenswrapper[5444]: W1101 20:48:34.465393 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9172d606_b0ef_4932_b501_9cb97c191c92.slice/crio-15a989f7b9b29d1f2ae90bf5cd04cb97a8917e93b012f2a4f5102c877696f2e2 WatchSource:0}: Error finding container 15a989f7b9b29d1f2ae90bf5cd04cb97a8917e93b012f2a4f5102c877696f2e2: Status 404 returned error can't find the container with id 15a989f7b9b29d1f2ae90bf5cd04cb97a8917e93b012f2a4f5102c877696f2e2 Nov 01 20:48:35.017578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:35.017549 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" event={"ID":"9172d606-b0ef-4932-b501-9cb97c191c92","Type":"ContainerStarted","Data":"6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14"} Nov 01 20:48:35.018005 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:35.017602 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" event={"ID":"9172d606-b0ef-4932-b501-9cb97c191c92","Type":"ContainerStarted","Data":"15a989f7b9b29d1f2ae90bf5cd04cb97a8917e93b012f2a4f5102c877696f2e2"} Nov 01 20:48:35.018005 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:35.017665 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:48:35.131313 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:35.131272 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" podStartSLOduration=2.131261546 podStartE2EDuration="2.131261546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:48:35.130794622 +0000 UTC m=+2716.075632490" watchObservedRunningTime="2026-11-01 20:48:35.131261546 +0000 UTC m=+2716.076099414" Nov 01 20:48:46.025039 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:48:46.025009 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:49:00.036518 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:49:00.036480 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:49:30.033691 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:49:30.033653 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:50:00.034150 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:50:00.034071 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:50:30.034969 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:50:30.034932 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:50:54.409823 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:54.409780 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr"] Nov 01 20:50:54.410317 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:54.410030 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" containerID="cri-o://9a2c33e8f1f57ea465d2600c5b175575e0fbe405ba0a16e009458e13fde18c7e" gracePeriod=30 Nov 01 20:50:57.876180 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:57.876145 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4ghvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:50:57.876630 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:57.876217 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:50:59.594580 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.594545 5444 generic.go:356] "Generic (PLEG): container finished" podID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerID="9a2c33e8f1f57ea465d2600c5b175575e0fbe405ba0a16e009458e13fde18c7e" exitCode=0 Nov 01 20:50:59.595032 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.594617 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" event={"ID":"4abf0006-4b7d-4789-9417-ae3b2bec1dc5","Type":"ContainerDied","Data":"9a2c33e8f1f57ea465d2600c5b175575e0fbe405ba0a16e009458e13fde18c7e"} Nov 01 20:50:59.668566 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.668538 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:50:59.760348 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760321 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760474 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760378 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760474 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760408 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760474 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760455 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760501 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760530 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760560 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760589 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760667 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760614 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket\") pod \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\" (UID: \"4abf0006-4b7d-4789-9417-ae3b2bec1dc5\") " Nov 01 20:50:59.760939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760767 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:59.760939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760827 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:59.760939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760853 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:59.760939 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760930 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.761152 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760948 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.761152 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.760981 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:59.761280 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.761253 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:50:59.762565 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.762543 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "kube-api-access-bhk95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:59.762706 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.762687 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:50:59.762850 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.762833 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:50:59.762989 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.762947 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy" pod "4abf0006-4b7d-4789-9417-ae3b2bec1dc5" (UID: "4abf0006-4b7d-4789-9417-ae3b2bec1dc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:50:59.861603 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861579 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.861603 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861602 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhk95\" (UniqueName: \"kubernetes.io/projected/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-kube-api-access-bhk95\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.861758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861614 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.861758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861622 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.861758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861631 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.861758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861653 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:50:59.861758 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:50:59.861661 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abf0006-4b7d-4789-9417-ae3b2bec1dc5-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:00.033911 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:51:00.033830 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:51:00.600126 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:00.600088 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" event={"ID":"4abf0006-4b7d-4789-9417-ae3b2bec1dc5","Type":"ContainerDied","Data":"346c327f77a1b7a09652a87feb7a22db7cddd7ada32aa2bb9b6aa3f8cf57a586"} Nov 01 20:51:00.600126 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:00.600107 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr" Nov 01 20:51:00.600619 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:00.600148 5444 scope.go:122] "RemoveContainer" containerID="9a2c33e8f1f57ea465d2600c5b175575e0fbe405ba0a16e009458e13fde18c7e" Nov 01 20:51:00.811189 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:00.811163 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr"] Nov 01 20:51:00.823342 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:00.823318 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4ghvr"] Nov 01 20:51:01.158683 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:01.158631 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" path="/var/lib/kubelet/pods/4abf0006-4b7d-4789-9417-ae3b2bec1dc5/volumes" Nov 01 20:51:09.955338 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:09.955306 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54"] Nov 01 20:51:14.956417 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:14.956362 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" podUID="9172d606-b0ef-4932-b501-9cb97c191c92" containerName="payload-processing" containerID="cri-o://6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14" gracePeriod=25 Nov 01 20:51:15.195781 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.195753 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:51:15.279051 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279030 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr\") pod \"9172d606-b0ef-4932-b501-9cb97c191c92\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " Nov 01 20:51:15.279155 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279061 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials\") pod \"9172d606-b0ef-4932-b501-9cb97c191c92\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " Nov 01 20:51:15.279155 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279095 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume\") pod \"9172d606-b0ef-4932-b501-9cb97c191c92\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " Nov 01 20:51:15.279274 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279176 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume\") pod \"9172d606-b0ef-4932-b501-9cb97c191c92\" (UID: \"9172d606-b0ef-4932-b501-9cb97c191c92\") " Nov 01 20:51:15.279934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279498 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume" pod "9172d606-b0ef-4932-b501-9cb97c191c92" (UID: "9172d606-b0ef-4932-b501-9cb97c191c92"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:15.279934 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279583 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume" pod "9172d606-b0ef-4932-b501-9cb97c191c92" (UID: "9172d606-b0ef-4932-b501-9cb97c191c92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:51:15.280108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.279986 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:15.280108 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.280008 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9172d606-b0ef-4932-b501-9cb97c191c92-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:15.281376 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.281349 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr" pod "9172d606-b0ef-4932-b501-9cb97c191c92" (UID: "9172d606-b0ef-4932-b501-9cb97c191c92"). InnerVolumeSpecName "kube-api-access-b7qbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:15.281459 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.281357 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials" pod "9172d606-b0ef-4932-b501-9cb97c191c92" (UID: "9172d606-b0ef-4932-b501-9cb97c191c92"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:51:15.380937 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.380911 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qbr\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-kube-api-access-b7qbr\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:15.380937 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.380935 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9172d606-b0ef-4932-b501-9cb97c191c92-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:51:15.660578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.660494 5444 generic.go:356] "Generic (PLEG): container finished" podID="9172d606-b0ef-4932-b501-9cb97c191c92" containerID="6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14" exitCode=0 Nov 01 20:51:15.660578 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.660542 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" Nov 01 20:51:15.660794 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.660583 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" event={"ID":"9172d606-b0ef-4932-b501-9cb97c191c92","Type":"ContainerDied","Data":"6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14"} Nov 01 20:51:15.660794 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.660656 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54" event={"ID":"9172d606-b0ef-4932-b501-9cb97c191c92","Type":"ContainerDied","Data":"15a989f7b9b29d1f2ae90bf5cd04cb97a8917e93b012f2a4f5102c877696f2e2"} Nov 01 20:51:15.660794 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.660683 5444 scope.go:122] "RemoveContainer" containerID="6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14" Nov 01 20:51:15.672061 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.672041 5444 scope.go:122] "RemoveContainer" containerID="6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14" Nov 01 20:51:15.672361 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:51:15.672339 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14\": container with ID starting with 6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14 not found: ID does not exist" containerID="6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14" Nov 01 20:51:15.672452 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.672375 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14"} err="failed to get container status \"6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14\": rpc error: code = NotFound desc = could not find container \"6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14\": container with ID starting with 6e5386d74d24328d458f18d79c505ca5a2d9dd6220b5b185be5f488359c87d14 not found: ID does not exist" Nov 01 20:51:15.822584 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.822555 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54"] Nov 01 20:51:15.824305 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:15.824274 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-6db8f5j54"] Nov 01 20:51:17.156923 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:51:17.156888 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9172d606-b0ef-4932-b501-9cb97c191c92" path="/var/lib/kubelet/pods/9172d606-b0ef-4932-b501-9cb97c191c92/volumes" Nov 01 20:51:30.035591 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:51:30.035551 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:52:00.035774 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:52:00.035724 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:52:30.034482 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:52:30.034447 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:53:00.033763 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:53:00.033689 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:53:30.034535 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:53:30.034498 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:53:55.751158 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:53:55.751116 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl"] Nov 01 20:53:55.751584 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:53:55.751376 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" containerID="cri-o://32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7" gracePeriod=30 Nov 01 20:53:57.779533 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:53:57.779495 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-r2grl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:53:57.779944 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:53:57.779561 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:54:00.034005 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:54:00.033971 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:54:00.986911 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:00.986881 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:54:01.130124 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130053 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130124 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130112 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130130 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130152 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130197 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130214 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130369 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130421 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130450 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl\") pod \"8d765043-d698-444c-8dc6-1d891c364570\" (UID: \"8d765043-d698-444c-8dc6-1d891c364570\") " Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130489 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130605 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:01.130796 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130630 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:01.131352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130804 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.131352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130825 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.131352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130841 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.131352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130869 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:54:01.131352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.130902 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:01.132400 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.132353 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:54:01.132509 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.132434 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:54:01.132509 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.132471 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:01.132604 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.132533 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl" pod "8d765043-d698-444c-8dc6-1d891c364570" (UID: "8d765043-d698-444c-8dc6-1d891c364570"). InnerVolumeSpecName "kube-api-access-nzfzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:54:01.231547 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.231512 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.231547 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.231545 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.231547 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.231559 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d765043-d698-444c-8dc6-1d891c364570-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.231822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.231572 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d765043-d698-444c-8dc6-1d891c364570-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.231822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.231583 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d765043-d698-444c-8dc6-1d891c364570-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.231822 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.231595 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzfzl\" (UniqueName: \"kubernetes.io/projected/8d765043-d698-444c-8dc6-1d891c364570-kube-api-access-nzfzl\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 20:54:01.323593 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.323559 5444 generic.go:356] "Generic (PLEG): container finished" podID="8d765043-d698-444c-8dc6-1d891c364570" containerID="32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7" exitCode=0 Nov 01 20:54:01.323821 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.323616 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" Nov 01 20:54:01.323821 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.323657 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" event={"ID":"8d765043-d698-444c-8dc6-1d891c364570","Type":"ContainerDied","Data":"32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7"} Nov 01 20:54:01.323821 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.323718 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl" event={"ID":"8d765043-d698-444c-8dc6-1d891c364570","Type":"ContainerDied","Data":"0927f89873c06983679cfec1f022f3df97785173d237022f861e5b7020caebb3"} Nov 01 20:54:01.323821 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.323740 5444 scope.go:122] "RemoveContainer" containerID="32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7" Nov 01 20:54:01.331972 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.331954 5444 scope.go:122] "RemoveContainer" containerID="32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7" Nov 01 20:54:01.332272 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:54:01.332251 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7\": container with ID starting with 32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7 not found: ID does not exist" containerID="32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7" Nov 01 20:54:01.332352 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.332293 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7"} err="failed to get container status \"32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7\": rpc error: code = NotFound desc = could not find container \"32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7\": container with ID starting with 32df2473dd7977fca69b9f5c0ae6ae110206d8b0a14991ff62ad78830d3d07c7 not found: ID does not exist" Nov 01 20:54:01.484940 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.484869 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl"] Nov 01 20:54:01.487556 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:01.487536 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-r2grl"] Nov 01 20:54:03.156694 ip-10-0-138-177 kubenswrapper[5444]: I1101 20:54:03.156662 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d765043-d698-444c-8dc6-1d891c364570" path="/var/lib/kubelet/pods/8d765043-d698-444c-8dc6-1d891c364570/volumes" Nov 01 20:54:30.033081 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:54:30.033034 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:55:00.034010 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:55:00.033975 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:55:30.034419 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:55:30.034379 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:56:00.033214 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:56:00.033134 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:56:30.034776 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:56:30.034724 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:57:00.035824 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:57:00.035786 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:57:30.033655 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:57:30.033600 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:58:00.035100 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:58:00.034998 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:58:30.038268 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:58:30.038228 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:59:00.033671 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:59:00.033577 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 20:59:30.034874 ip-10-0-138-177 kubenswrapper[5444]: E1101 20:59:30.034836 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:00:00.034689 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:00.034649 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:00:00.227812 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.227775 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl"] Nov 01 21:00:00.228132 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228116 5444 cpu_manager.go:414] "containerMap: removing container" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" Nov 01 21:00:00.228132 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228132 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" Nov 01 21:00:00.228227 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228156 5444 cpu_manager.go:414] "containerMap: removing container" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" Nov 01 21:00:00.228227 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228161 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" Nov 01 21:00:00.228227 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228172 5444 cpu_manager.go:414] "containerMap: removing container" podUID="9172d606-b0ef-4932-b501-9cb97c191c92" containerName="payload-processing" Nov 01 21:00:00.228227 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228178 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9172d606-b0ef-4932-b501-9cb97c191c92" containerName="payload-processing" Nov 01 21:00:00.228227 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228223 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d765043-d698-444c-8dc6-1d891c364570" containerName="istio-proxy" Nov 01 21:00:00.228413 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228232 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="9172d606-b0ef-4932-b501-9cb97c191c92" containerName="payload-processing" Nov 01 21:00:00.228413 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.228239 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abf0006-4b7d-4789-9417-ae3b2bec1dc5" containerName="istio-proxy" Nov 01 21:00:00.230969 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.230953 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.283350 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.283327 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:00:00.337175 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.337081 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl"] Nov 01 21:00:00.381200 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.381169 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.381378 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.381211 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46km\" (UniqueName: \"kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.482292 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.482257 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.482476 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.482306 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46km\" (UniqueName: \"kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.482950 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.482929 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.538516 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.538486 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46km\" (UniqueName: \"kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km\") pod \"maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.543263 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.543245 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:00.910713 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:00.910596 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl"] Nov 01 21:00:00.914525 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:00:00.914502 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89565a31_dba9_4af9_a5e3_a07891680800.slice/crio-e1735e228a137a2e052570595f77157edf96c715218d527ba23fd2f0f3f57803 WatchSource:0}: Error finding container e1735e228a137a2e052570595f77157edf96c715218d527ba23fd2f0f3f57803: Status 404 returned error can't find the container with id e1735e228a137a2e052570595f77157edf96c715218d527ba23fd2f0f3f57803 Nov 01 21:00:01.741939 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:01.741902 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerStarted","Data":"bc329aa5e4c90c809e1d82cff906c308e98f9d5c7ec45b65e005368507e77afc"} Nov 01 21:00:01.742372 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:01.741967 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerStarted","Data":"e1735e228a137a2e052570595f77157edf96c715218d527ba23fd2f0f3f57803"} Nov 01 21:00:01.775551 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:01.775522 5444 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89565a31_dba9_4af9_a5e3_a07891680800.slice/crio-bc329aa5e4c90c809e1d82cff906c308e98f9d5c7ec45b65e005368507e77afc.scope\": RecentStats: unable to find data in memory cache]" Nov 01 21:00:01.853015 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:01.852961 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" podStartSLOduration=1.852947908 podStartE2EDuration="1.852947908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:01.852851526 +0000 UTC m=+3402.797689394" watchObservedRunningTime="2026-11-01 21:00:01.852947908 +0000 UTC m=+3402.797785775" Nov 01 21:00:02.747247 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:02.747212 5444 generic.go:356] "Generic (PLEG): container finished" podID="89565a31-dba9-4af9-a5e3-a07891680800" containerID="bc329aa5e4c90c809e1d82cff906c308e98f9d5c7ec45b65e005368507e77afc" exitCode=60 Nov 01 21:00:02.747697 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:02.747276 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerDied","Data":"bc329aa5e4c90c809e1d82cff906c308e98f9d5c7ec45b65e005368507e77afc"} Nov 01 21:00:02.747697 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:02.747515 5444 scope.go:122] "RemoveContainer" containerID="bc329aa5e4c90c809e1d82cff906c308e98f9d5c7ec45b65e005368507e77afc" Nov 01 21:00:03.752901 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:03.752864 5444 generic.go:356] "Generic (PLEG): container finished" podID="89565a31-dba9-4af9-a5e3-a07891680800" containerID="22deec909a365af47d41ed041d7df34a1762d1e9692eca7fc041b30c62f9023c" exitCode=60 Nov 01 21:00:03.753403 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:03.752920 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerDied","Data":"22deec909a365af47d41ed041d7df34a1762d1e9692eca7fc041b30c62f9023c"} Nov 01 21:00:03.753403 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:03.752967 5444 scope.go:122] "RemoveContainer" containerID="bc329aa5e4c90c809e1d82cff906c308e98f9d5c7ec45b65e005368507e77afc" Nov 01 21:00:03.753403 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:03.753214 5444 scope.go:122] "RemoveContainer" containerID="22deec909a365af47d41ed041d7df34a1762d1e9692eca7fc041b30c62f9023c" Nov 01 21:00:03.753566 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:03.753451 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl_odh-ai-gateway-infra(89565a31-dba9-4af9-a5e3-a07891680800)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" podUID="89565a31-dba9-4af9-a5e3-a07891680800" Nov 01 21:00:04.758122 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:04.758101 5444 scope.go:122] "RemoveContainer" containerID="22deec909a365af47d41ed041d7df34a1762d1e9692eca7fc041b30c62f9023c" Nov 01 21:00:04.758478 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:04.758291 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl_odh-ai-gateway-infra(89565a31-dba9-4af9-a5e3-a07891680800)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" podUID="89565a31-dba9-4af9-a5e3-a07891680800" Nov 01 21:00:18.151824 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:18.151796 5444 scope.go:122] "RemoveContainer" containerID="22deec909a365af47d41ed041d7df34a1762d1e9692eca7fc041b30c62f9023c" Nov 01 21:00:18.813313 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:18.813282 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerStarted","Data":"aeb6d6957d3cca7b1a14ce13cc3ce563b7317b6b9067f0b26368e1c808804d51"} Nov 01 21:00:19.819204 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:19.819133 5444 generic.go:356] "Generic (PLEG): container finished" podID="89565a31-dba9-4af9-a5e3-a07891680800" containerID="aeb6d6957d3cca7b1a14ce13cc3ce563b7317b6b9067f0b26368e1c808804d51" exitCode=60 Nov 01 21:00:19.819667 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:19.819205 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerDied","Data":"aeb6d6957d3cca7b1a14ce13cc3ce563b7317b6b9067f0b26368e1c808804d51"} Nov 01 21:00:19.819667 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:19.819263 5444 scope.go:122] "RemoveContainer" containerID="22deec909a365af47d41ed041d7df34a1762d1e9692eca7fc041b30c62f9023c" Nov 01 21:00:19.819667 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:19.819547 5444 scope.go:122] "RemoveContainer" containerID="aeb6d6957d3cca7b1a14ce13cc3ce563b7317b6b9067f0b26368e1c808804d51" Nov 01 21:00:19.819860 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:19.819791 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl_odh-ai-gateway-infra(89565a31-dba9-4af9-a5e3-a07891680800)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" podUID="89565a31-dba9-4af9-a5e3-a07891680800" Nov 01 21:00:19.943994 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:19.943969 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl"] Nov 01 21:00:20.952206 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:20.952184 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:21.052927 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.052905 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km\" (UniqueName: \"kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km\") pod \"89565a31-dba9-4af9-a5e3-a07891680800\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " Nov 01 21:00:21.053047 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.052963 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca\") pod \"89565a31-dba9-4af9-a5e3-a07891680800\" (UID: \"89565a31-dba9-4af9-a5e3-a07891680800\") " Nov 01 21:00:21.053310 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.053288 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca" pod "89565a31-dba9-4af9-a5e3-a07891680800" (UID: "89565a31-dba9-4af9-a5e3-a07891680800"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:21.054818 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.054797 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km" pod "89565a31-dba9-4af9-a5e3-a07891680800" (UID: "89565a31-dba9-4af9-a5e3-a07891680800"). InnerVolumeSpecName "kube-api-access-j46km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:21.153656 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.153582 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89565a31-dba9-4af9-a5e3-a07891680800-maas-api-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.153656 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.153606 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46km\" (UniqueName: \"kubernetes.io/projected/89565a31-dba9-4af9-a5e3-a07891680800-kube-api-access-j46km\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:21.829961 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.829922 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" event={"ID":"89565a31-dba9-4af9-a5e3-a07891680800","Type":"ContainerDied","Data":"e1735e228a137a2e052570595f77157edf96c715218d527ba23fd2f0f3f57803"} Nov 01 21:00:21.830137 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.830009 5444 scope.go:122] "RemoveContainer" containerID="aeb6d6957d3cca7b1a14ce13cc3ce563b7317b6b9067f0b26368e1c808804d51" Nov 01 21:00:21.830137 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.829935 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl" Nov 01 21:00:21.989726 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.989697 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl"] Nov 01 21:00:21.997887 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:21.997856 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-99d504-29892780-26qxl"] Nov 01 21:00:23.156859 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:23.156832 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89565a31-dba9-4af9-a5e3-a07891680800" path="/var/lib/kubelet/pods/89565a31-dba9-4af9-a5e3-a07891680800/volumes" Nov 01 21:00:30.033024 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:30.032986 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:00:46.091964 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.091928 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz"] Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092222 5444 cpu_manager.go:414] "containerMap: removing container" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092233 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092252 5444 cpu_manager.go:414] "containerMap: removing container" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092259 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092267 5444 cpu_manager.go:414] "containerMap: removing container" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092271 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092335 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.092466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.092344 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:00:46.095136 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.095120 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.123743 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.123715 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.123869 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.123746 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79g87\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.123869 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.123833 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.123948 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.123875 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.147358 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.147328 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-433dc433b175\"" Nov 01 21:00:46.147358 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.147354 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cfl-224e35-provider-cred\"" Nov 01 21:00:46.197129 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.197098 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 21:00:46.197129 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.197099 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-1c1bf2169d38\"" Nov 01 21:00:46.197129 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.197098 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 21:00:46.197129 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.197099 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dhzm4\"" Nov 01 21:00:46.197129 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.197112 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 21:00:46.197450 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.197170 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 21:00:46.198104 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.198081 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz"] Nov 01 21:00:46.224302 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.224271 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.224461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.224311 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79g87\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.224461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.224377 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.224461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.224405 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.225247 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.225218 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.225361 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.225295 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.226967 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.226945 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.281079 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.281049 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79g87\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.407461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.407367 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:46.776685 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.776656 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz"] Nov 01 21:00:46.780463 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:00:46.780434 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9f2ec3_f58e_4ef4_9fff_575d708c0efc.slice/crio-cfb9de35f523b897f307b74fb00f0391b199a268df3ef8c6163f3d28e8677a18 WatchSource:0}: Error finding container cfb9de35f523b897f307b74fb00f0391b199a268df3ef8c6163f3d28e8677a18: Status 404 returned error can't find the container with id cfb9de35f523b897f307b74fb00f0391b199a268df3ef8c6163f3d28e8677a18 Nov 01 21:00:46.933448 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.933411 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" event={"ID":"df9f2ec3-f58e-4ef4-9fff-575d708c0efc","Type":"ContainerStarted","Data":"7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e"} Nov 01 21:00:46.933604 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.933472 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" event={"ID":"df9f2ec3-f58e-4ef4-9fff-575d708c0efc","Type":"ContainerStarted","Data":"cfb9de35f523b897f307b74fb00f0391b199a268df3ef8c6163f3d28e8677a18"} Nov 01 21:00:46.933604 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:46.933510 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:47.045993 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:47.045898 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" podStartSLOduration=1.045885244 podStartE2EDuration="1.045885244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:47.045190996 +0000 UTC m=+3447.990028867" watchObservedRunningTime="2026-11-01 21:00:47.045885244 +0000 UTC m=+3447.990723111" Nov 01 21:00:48.035066 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:48.035031 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-433dc433b175: secret "agc-cred-433dc433b175" not found Nov 01 21:00:48.035066 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:48.035062 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz: secret "agc-cred-433dc433b175" not found Nov 01 21:00:48.035525 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:48.035121 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials podName:df9f2ec3-f58e-4ef4-9fff-575d708c0efc nodeName:}" failed. No retries permitted until 2026-11-01 21:00:48.535103977 +0000 UTC m=+3449.479941839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc") : secret "agc-cred-433dc433b175" not found Nov 01 21:00:48.538750 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:48.538720 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-433dc433b175: secret "agc-cred-433dc433b175" not found Nov 01 21:00:48.538750 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:48.538748 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz: secret "agc-cred-433dc433b175" not found Nov 01 21:00:48.538970 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:48.538797 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials podName:df9f2ec3-f58e-4ef4-9fff-575d708c0efc nodeName:}" failed. No retries permitted until 2026-11-01 21:00:49.538784087 +0000 UTC m=+3450.483621946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc") : secret "agc-cred-433dc433b175" not found Nov 01 21:00:48.564035 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:48.564008 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz"] Nov 01 21:00:49.544047 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:49.544012 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-433dc433b175: secret "agc-cred-433dc433b175" not found Nov 01 21:00:49.544674 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:49.544631 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz: secret "agc-cred-433dc433b175" not found Nov 01 21:00:49.544797 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:49.544740 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials podName:df9f2ec3-f58e-4ef4-9fff-575d708c0efc nodeName:}" failed. No retries permitted until 2026-11-01 21:00:51.54470872 +0000 UTC m=+3452.489546591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc") : secret "agc-cred-433dc433b175" not found Nov 01 21:00:51.559712 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:51.559678 5444 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-99d504/agc-cred-433dc433b175: secret "agc-cred-433dc433b175" not found Nov 01 21:00:51.559712 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:51.559711 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz: secret "agc-cred-433dc433b175" not found Nov 01 21:00:51.560231 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:51.559766 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials podName:df9f2ec3-f58e-4ef4-9fff-575d708c0efc nodeName:}" failed. No retries permitted until 2026-11-01 21:00:55.55974992 +0000 UTC m=+3456.504587782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc") : secret "agc-cred-433dc433b175" not found Nov 01 21:00:53.941315 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:53.941273 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" podUID="df9f2ec3-f58e-4ef4-9fff-575d708c0efc" containerName="payload-processing" containerID="cri-o://7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e" gracePeriod=25 Nov 01 21:00:54.180453 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.180430 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:54.278383 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.278358 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials\") pod \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " Nov 01 21:00:54.278531 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.278393 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume\") pod \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " Nov 01 21:00:54.278531 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.278427 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume\") pod \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " Nov 01 21:00:54.278531 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.278454 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87\") pod \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\" (UID: \"df9f2ec3-f58e-4ef4-9fff-575d708c0efc\") " Nov 01 21:00:54.278808 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.278785 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume" pod "df9f2ec3-f58e-4ef4-9fff-575d708c0efc" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:54.278862 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.278833 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume" pod "df9f2ec3-f58e-4ef4-9fff-575d708c0efc" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:00:54.280474 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.280446 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87" pod "df9f2ec3-f58e-4ef4-9fff-575d708c0efc" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc"). InnerVolumeSpecName "kube-api-access-79g87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:54.280557 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.280539 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials" pod "df9f2ec3-f58e-4ef4-9fff-575d708c0efc" (UID: "df9f2ec3-f58e-4ef4-9fff-575d708c0efc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:00:54.379298 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.379267 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79g87\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-kube-api-access-79g87\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:54.379298 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.379293 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:54.379298 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.379302 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:54.379514 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.379312 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9f2ec3-f58e-4ef4-9fff-575d708c0efc-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:00:54.966884 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.966847 5444 generic.go:356] "Generic (PLEG): container finished" podID="df9f2ec3-f58e-4ef4-9fff-575d708c0efc" containerID="7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e" exitCode=0 Nov 01 21:00:54.967307 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.966899 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" event={"ID":"df9f2ec3-f58e-4ef4-9fff-575d708c0efc","Type":"ContainerDied","Data":"7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e"} Nov 01 21:00:54.967307 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.966940 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" event={"ID":"df9f2ec3-f58e-4ef4-9fff-575d708c0efc","Type":"ContainerDied","Data":"cfb9de35f523b897f307b74fb00f0391b199a268df3ef8c6163f3d28e8677a18"} Nov 01 21:00:54.967307 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.966964 5444 scope.go:122] "RemoveContainer" containerID="7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e" Nov 01 21:00:54.967307 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.966902 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz" Nov 01 21:00:54.976102 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.976084 5444 scope.go:122] "RemoveContainer" containerID="7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e" Nov 01 21:00:54.976356 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:00:54.976334 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e\": container with ID starting with 7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e not found: ID does not exist" containerID="7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e" Nov 01 21:00:54.976398 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:54.976364 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e"} err="failed to get container status \"7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e\": rpc error: code = NotFound desc = could not find container \"7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e\": container with ID starting with 7fcce87593c00a7d8d93df39eb5479d097ad158714b7695bcb3a3d514ec00d5e not found: ID does not exist" Nov 01 21:00:55.127717 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:55.127693 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz"] Nov 01 21:00:55.176952 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:55.176926 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-845fvpxsz"] Nov 01 21:00:57.157107 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:00:57.157075 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9f2ec3-f58e-4ef4-9fff-575d708c0efc" path="/var/lib/kubelet/pods/df9f2ec3-f58e-4ef4-9fff-575d708c0efc/volumes" Nov 01 21:01:00.034347 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:01:00.034314 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:01:02.199821 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.199788 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg"] Nov 01 21:01:02.200229 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.200105 5444 cpu_manager.go:414] "containerMap: removing container" podUID="df9f2ec3-f58e-4ef4-9fff-575d708c0efc" containerName="payload-processing" Nov 01 21:01:02.200229 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.200117 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9f2ec3-f58e-4ef4-9fff-575d708c0efc" containerName="payload-processing" Nov 01 21:01:02.200229 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.200184 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="89565a31-dba9-4af9-a5e3-a07891680800" containerName="cleanup" Nov 01 21:01:02.200229 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.200192 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9f2ec3-f58e-4ef4-9fff-575d708c0efc" containerName="payload-processing" Nov 01 21:01:02.203108 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.203092 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.236859 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.236837 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.236989 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.236874 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pp4\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.236989 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.236902 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.237141 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.237128 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.304572 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.304541 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-dhzm4\"" Nov 01 21:01:02.304572 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.304562 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 21:01:02.304782 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.304564 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 21:01:02.304782 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.304564 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"e2e-cfl-224e35-provider-cred\"" Nov 01 21:01:02.304782 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.304597 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 21:01:02.304782 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.304546 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 21:01:02.305464 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.305446 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg"] Nov 01 21:01:02.338073 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.338050 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.338213 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.338101 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.338213 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.338130 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.338213 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.338161 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pp4\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.338856 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.338836 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.338948 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.338856 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.340546 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.340525 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.394456 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.394420 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pp4\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.514455 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.514433 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:02.884368 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:02.884274 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg"] Nov 01 21:01:02.893319 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:01:02.893291 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3920e2e2_06f7_4541_ad32_189eb773ea50.slice/crio-318fc8c24690faa0fdae8ff13d731850672549d2a6beaaa40dd64b052fe63516 WatchSource:0}: Error finding container 318fc8c24690faa0fdae8ff13d731850672549d2a6beaaa40dd64b052fe63516: Status 404 returned error can't find the container with id 318fc8c24690faa0fdae8ff13d731850672549d2a6beaaa40dd64b052fe63516 Nov 01 21:01:03.001200 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:03.001173 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" event={"ID":"3920e2e2-06f7-4541-ad32-189eb773ea50","Type":"ContainerStarted","Data":"7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7"} Nov 01 21:01:03.001302 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:03.001220 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" event={"ID":"3920e2e2-06f7-4541-ad32-189eb773ea50","Type":"ContainerStarted","Data":"318fc8c24690faa0fdae8ff13d731850672549d2a6beaaa40dd64b052fe63516"} Nov 01 21:01:03.001347 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:03.001307 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:03.114171 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:03.114125 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" podStartSLOduration=1.114109879 podStartE2EDuration="1.114109879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:03.112781116 +0000 UTC m=+3464.057618984" watchObservedRunningTime="2026-11-01 21:01:03.114109879 +0000 UTC m=+3464.058947741" Nov 01 21:01:14.008156 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:14.008116 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:01:14.218311 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:14.218284 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg"] Nov 01 21:01:14.219369 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:14.219345 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:01:19.218548 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.218507 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerName="payload-processing" containerID="cri-o://7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7" gracePeriod=25 Nov 01 21:01:19.454942 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.454918 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:19.570038 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.570006 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4\") pod \"3920e2e2-06f7-4541-ad32-189eb773ea50\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " Nov 01 21:01:19.570198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.570086 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials\") pod \"3920e2e2-06f7-4541-ad32-189eb773ea50\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " Nov 01 21:01:19.570198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.570128 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume\") pod \"3920e2e2-06f7-4541-ad32-189eb773ea50\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " Nov 01 21:01:19.570198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.570183 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume\") pod \"3920e2e2-06f7-4541-ad32-189eb773ea50\" (UID: \"3920e2e2-06f7-4541-ad32-189eb773ea50\") " Nov 01 21:01:19.570842 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.570806 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume" pod "3920e2e2-06f7-4541-ad32-189eb773ea50" (UID: "3920e2e2-06f7-4541-ad32-189eb773ea50"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:01:19.570992 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.570936 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume" pod "3920e2e2-06f7-4541-ad32-189eb773ea50" (UID: "3920e2e2-06f7-4541-ad32-189eb773ea50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:01:19.572246 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.572216 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials" pod "3920e2e2-06f7-4541-ad32-189eb773ea50" (UID: "3920e2e2-06f7-4541-ad32-189eb773ea50"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:19.572370 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.572351 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4" pod "3920e2e2-06f7-4541-ad32-189eb773ea50" (UID: "3920e2e2-06f7-4541-ad32-189eb773ea50"). InnerVolumeSpecName "kube-api-access-67pp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:01:19.671204 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.671175 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:19.671204 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.671200 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3920e2e2-06f7-4541-ad32-189eb773ea50-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:19.671204 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.671210 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67pp4\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-kube-api-access-67pp4\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:19.671424 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:19.671219 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3920e2e2-06f7-4541-ad32-189eb773ea50-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:01:20.070108 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.070075 5444 generic.go:356] "Generic (PLEG): container finished" podID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerID="7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7" exitCode=0 Nov 01 21:01:20.070226 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.070113 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" event={"ID":"3920e2e2-06f7-4541-ad32-189eb773ea50","Type":"ContainerDied","Data":"7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7"} Nov 01 21:01:20.070226 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.070131 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" Nov 01 21:01:20.070226 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.070148 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg" event={"ID":"3920e2e2-06f7-4541-ad32-189eb773ea50","Type":"ContainerDied","Data":"318fc8c24690faa0fdae8ff13d731850672549d2a6beaaa40dd64b052fe63516"} Nov 01 21:01:20.070226 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.070175 5444 scope.go:122] "RemoveContainer" containerID="7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7" Nov 01 21:01:20.077984 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.077954 5444 scope.go:122] "RemoveContainer" containerID="7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7" Nov 01 21:01:20.078311 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:01:20.078287 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7\": container with ID starting with 7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7 not found: ID does not exist" containerID="7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7" Nov 01 21:01:20.078383 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.078322 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7"} err="failed to get container status \"7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7\": rpc error: code = NotFound desc = could not find container \"7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7\": container with ID starting with 7ab60bd775e41e4bd7af36270025443792e5d5893aeb3783280eab3557df7da7 not found: ID does not exist" Nov 01 21:01:20.231363 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.231340 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg"] Nov 01 21:01:20.280799 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:20.280750 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-5b8c7f4qg"] Nov 01 21:01:21.157001 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:01:21.156972 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" path="/var/lib/kubelet/pods/3920e2e2-06f7-4541-ad32-189eb773ea50/volumes" Nov 01 21:01:30.034149 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:01:30.034107 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:02:00.034554 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:00.034517 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:02:24.333041 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.333009 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89"] Nov 01 21:02:24.333613 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.333590 5444 cpu_manager.go:414] "containerMap: removing container" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerName="payload-processing" Nov 01 21:02:24.333722 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.333617 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerName="payload-processing" Nov 01 21:02:24.333781 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.333757 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="3920e2e2-06f7-4541-ad32-189eb773ea50" containerName="payload-processing" Nov 01 21:02:24.336801 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.336781 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.374217 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.374190 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.374316 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.374231 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.374316 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.374261 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.374398 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.374377 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5j8\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.389749 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.389731 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 21:02:24.439202 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.439181 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 21:02:24.439202 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.439180 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a4bbb12e7abe\"" Nov 01 21:02:24.439358 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.439185 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 21:02:24.439358 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.439258 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 21:02:24.439358 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.439324 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-w2d9g\"" Nov 01 21:02:24.440038 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.440023 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89"] Nov 01 21:02:24.475494 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.475476 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.475587 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.475504 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.475587 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.475528 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5j8\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.475587 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.475574 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.476202 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.476185 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.476317 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.476298 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.477756 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.477739 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.531372 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.531336 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5j8\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8\") pod \"payload-processing-external-model-7787bbd4d-nvm89\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:24.648541 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:24.648470 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:25.021475 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:25.021443 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89"] Nov 01 21:02:25.025270 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:02:25.025241 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f206bc_e3d8_4ed1_825e_a36bfc2ad20a.slice/crio-cb7d0a43c2a54e0040763356582f864b9d7b4bd7b62c9389ef9a5e3a454648ac WatchSource:0}: Error finding container cb7d0a43c2a54e0040763356582f864b9d7b4bd7b62c9389ef9a5e3a454648ac: Status 404 returned error can't find the container with id cb7d0a43c2a54e0040763356582f864b9d7b4bd7b62c9389ef9a5e3a454648ac Nov 01 21:02:25.322921 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:25.322829 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" event={"ID":"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a","Type":"ContainerStarted","Data":"0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974"} Nov 01 21:02:25.322921 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:25.322886 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" event={"ID":"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a","Type":"ContainerStarted","Data":"cb7d0a43c2a54e0040763356582f864b9d7b4bd7b62c9389ef9a5e3a454648ac"} Nov 01 21:02:25.322921 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:25.322910 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:25.447015 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:25.446966 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" podStartSLOduration=1.446953236 podStartE2EDuration="1.446953236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:25.446343559 +0000 UTC m=+3546.391181427" watchObservedRunningTime="2026-11-01 21:02:25.446953236 +0000 UTC m=+3546.391791083" Nov 01 21:02:26.391876 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:26.391841 5444 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a4bbb12e7abe: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:26.391876 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:26.391869 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:26.392344 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:26.391934 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials podName:92f206bc-e3d8-4ed1-825e-a36bfc2ad20a nodeName:}" failed. No retries permitted until 2026-11-01 21:02:26.891913621 +0000 UTC m=+3547.836751470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials") pod "payload-processing-external-model-7787bbd4d-nvm89" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a") : secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:26.895011 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:26.894980 5444 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a4bbb12e7abe: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:26.895011 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:26.895003 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:26.895424 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:26.895055 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials podName:92f206bc-e3d8-4ed1-825e-a36bfc2ad20a nodeName:}" failed. No retries permitted until 2026-11-01 21:02:27.89503959 +0000 UTC m=+3548.839877438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials") pod "payload-processing-external-model-7787bbd4d-nvm89" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a") : secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:27.755850 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:27.755817 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89"] Nov 01 21:02:27.902993 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:27.902967 5444 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a4bbb12e7abe: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:27.902993 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:27.902989 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:27.903399 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:27.903034 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials podName:92f206bc-e3d8-4ed1-825e-a36bfc2ad20a nodeName:}" failed. No retries permitted until 2026-11-01 21:02:29.903021106 +0000 UTC m=+3550.847858953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials") pod "payload-processing-external-model-7787bbd4d-nvm89" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a") : secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:29.920511 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:29.920479 5444 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a4bbb12e7abe: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:29.920511 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:29.920502 5444 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89: secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:29.920946 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:29.920549 5444 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials podName:92f206bc-e3d8-4ed1-825e-a36bfc2ad20a nodeName:}" failed. No retries permitted until 2026-11-01 21:02:33.920537426 +0000 UTC m=+3554.865375273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials") pod "payload-processing-external-model-7787bbd4d-nvm89" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a") : secret "agc-cred-a4bbb12e7abe" not found Nov 01 21:02:30.033909 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:30.033877 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:02:32.756947 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:32.756869 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" podUID="92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" containerName="payload-processing" containerID="cri-o://0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974" gracePeriod=25 Nov 01 21:02:32.999926 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:32.999903 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:33.052168 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.052098 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume\") pod \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " Nov 01 21:02:33.052168 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.052148 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials\") pod \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " Nov 01 21:02:33.052372 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.052178 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8\") pod \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " Nov 01 21:02:33.052372 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.052203 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume\") pod \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\" (UID: \"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a\") " Nov 01 21:02:33.052705 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.052547 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume" pod "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:33.052841 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.052767 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume" pod "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:33.054581 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.054548 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials" pod "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:33.054691 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.054612 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8" pod "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" (UID: "92f206bc-e3d8-4ed1-825e-a36bfc2ad20a"). InnerVolumeSpecName "kube-api-access-qk5j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:33.153228 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.153197 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:33.153343 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.153229 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:33.153343 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.153246 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5j8\" (UniqueName: \"kubernetes.io/projected/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-kube-api-access-qk5j8\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:33.153343 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.153260 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:33.357579 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.357514 5444 generic.go:356] "Generic (PLEG): container finished" podID="92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" containerID="0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974" exitCode=0 Nov 01 21:02:33.357579 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.357558 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" Nov 01 21:02:33.357761 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.357594 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" event={"ID":"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a","Type":"ContainerDied","Data":"0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974"} Nov 01 21:02:33.357761 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.357661 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89" event={"ID":"92f206bc-e3d8-4ed1-825e-a36bfc2ad20a","Type":"ContainerDied","Data":"cb7d0a43c2a54e0040763356582f864b9d7b4bd7b62c9389ef9a5e3a454648ac"} Nov 01 21:02:33.357761 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.357685 5444 scope.go:122] "RemoveContainer" containerID="0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974" Nov 01 21:02:33.365310 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.365286 5444 scope.go:122] "RemoveContainer" containerID="0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974" Nov 01 21:02:33.365573 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:33.365556 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974\": container with ID starting with 0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974 not found: ID does not exist" containerID="0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974" Nov 01 21:02:33.365690 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.365583 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974"} err="failed to get container status \"0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974\": rpc error: code = NotFound desc = could not find container \"0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974\": container with ID starting with 0ea545cfb2ea19661c740f9138a5f34d6d696ca418fb5d284010d24b18e15974 not found: ID does not exist" Nov 01 21:02:33.518064 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.518039 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89"] Nov 01 21:02:33.567279 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:33.567258 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-nvm89"] Nov 01 21:02:34.425984 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.425959 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4"] Nov 01 21:02:34.426378 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.426250 5444 cpu_manager.go:414] "containerMap: removing container" podUID="92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" containerName="payload-processing" Nov 01 21:02:34.426378 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.426260 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" containerName="payload-processing" Nov 01 21:02:34.426378 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.426321 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" containerName="payload-processing" Nov 01 21:02:34.430413 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.430396 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.487187 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.487166 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a4bbb12e7abe\"" Nov 01 21:02:34.487623 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.487605 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lnrz6\"" Nov 01 21:02:34.488273 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.488255 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4"] Nov 01 21:02:34.536440 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.536422 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 21:02:34.536440 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.536432 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 21:02:34.536709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.536694 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 21:02:34.536709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.536707 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 21:02:34.564592 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.564573 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.564704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.564618 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.564704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.564653 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wb6\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.564809 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.564792 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.666139 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.666112 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.666226 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.666160 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.666226 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.666187 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wb6\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.666309 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.666239 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.666755 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.666734 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.666941 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.666923 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.668488 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.668469 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.722042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.721992 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wb6\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6\") pod \"payload-processing-external-model-7787bbd4d-p6kg4\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:34.742812 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:34.742789 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:35.125625 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:35.125586 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4"] Nov 01 21:02:35.130057 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:02:35.130026 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b915478_6c59_48fe_852c_16ab3f8ccee5.slice/crio-5e4bf831372310994caa1a67ee127e0bb27b243b701738cbd2bfbbe921e38cfc WatchSource:0}: Error finding container 5e4bf831372310994caa1a67ee127e0bb27b243b701738cbd2bfbbe921e38cfc: Status 404 returned error can't find the container with id 5e4bf831372310994caa1a67ee127e0bb27b243b701738cbd2bfbbe921e38cfc Nov 01 21:02:35.157023 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:35.157001 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f206bc-e3d8-4ed1-825e-a36bfc2ad20a" path="/var/lib/kubelet/pods/92f206bc-e3d8-4ed1-825e-a36bfc2ad20a/volumes" Nov 01 21:02:35.368709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:35.368672 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" event={"ID":"8b915478-6c59-48fe-852c-16ab3f8ccee5","Type":"ContainerStarted","Data":"be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8"} Nov 01 21:02:35.368877 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:35.368728 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" event={"ID":"8b915478-6c59-48fe-852c-16ab3f8ccee5","Type":"ContainerStarted","Data":"5e4bf831372310994caa1a67ee127e0bb27b243b701738cbd2bfbbe921e38cfc"} Nov 01 21:02:35.368877 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:35.368749 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:35.482368 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:35.482242 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" podStartSLOduration=1.482228428 podStartE2EDuration="1.482228428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:35.48142258 +0000 UTC m=+3556.426260447" watchObservedRunningTime="2026-11-01 21:02:35.482228428 +0000 UTC m=+3556.427066295" Nov 01 21:02:38.690150 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:38.690117 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4"] Nov 01 21:02:43.691030 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:43.690967 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" podUID="8b915478-6c59-48fe-852c-16ab3f8ccee5" containerName="payload-processing" containerID="cri-o://be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8" gracePeriod=25 Nov 01 21:02:43.933900 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:43.933875 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:44.044486 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.044465 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume\") pod \"8b915478-6c59-48fe-852c-16ab3f8ccee5\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " Nov 01 21:02:44.044610 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.044495 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials\") pod \"8b915478-6c59-48fe-852c-16ab3f8ccee5\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " Nov 01 21:02:44.044610 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.044539 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume\") pod \"8b915478-6c59-48fe-852c-16ab3f8ccee5\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " Nov 01 21:02:44.044610 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.044563 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6\") pod \"8b915478-6c59-48fe-852c-16ab3f8ccee5\" (UID: \"8b915478-6c59-48fe-852c-16ab3f8ccee5\") " Nov 01 21:02:44.044961 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.044938 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume" pod "8b915478-6c59-48fe-852c-16ab3f8ccee5" (UID: "8b915478-6c59-48fe-852c-16ab3f8ccee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:44.045035 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.044932 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume" pod "8b915478-6c59-48fe-852c-16ab3f8ccee5" (UID: "8b915478-6c59-48fe-852c-16ab3f8ccee5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:02:44.046539 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.046506 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6" pod "8b915478-6c59-48fe-852c-16ab3f8ccee5" (UID: "8b915478-6c59-48fe-852c-16ab3f8ccee5"). InnerVolumeSpecName "kube-api-access-g9wb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:44.046539 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.046534 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials" pod "8b915478-6c59-48fe-852c-16ab3f8ccee5" (UID: "8b915478-6c59-48fe-852c-16ab3f8ccee5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:02:44.145214 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.145186 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:44.145214 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.145213 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:44.145356 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.145226 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b915478-6c59-48fe-852c-16ab3f8ccee5-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:44.145356 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.145239 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wb6\" (UniqueName: \"kubernetes.io/projected/8b915478-6c59-48fe-852c-16ab3f8ccee5-kube-api-access-g9wb6\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:02:44.409704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.409603 5444 generic.go:356] "Generic (PLEG): container finished" podID="8b915478-6c59-48fe-852c-16ab3f8ccee5" containerID="be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8" exitCode=0 Nov 01 21:02:44.409704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.409657 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" Nov 01 21:02:44.409920 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.409713 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" event={"ID":"8b915478-6c59-48fe-852c-16ab3f8ccee5","Type":"ContainerDied","Data":"be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8"} Nov 01 21:02:44.409920 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.409759 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4" event={"ID":"8b915478-6c59-48fe-852c-16ab3f8ccee5","Type":"ContainerDied","Data":"5e4bf831372310994caa1a67ee127e0bb27b243b701738cbd2bfbbe921e38cfc"} Nov 01 21:02:44.409920 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.409780 5444 scope.go:122] "RemoveContainer" containerID="be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8" Nov 01 21:02:44.418585 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.418569 5444 scope.go:122] "RemoveContainer" containerID="be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8" Nov 01 21:02:44.418848 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:02:44.418825 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8\": container with ID starting with be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8 not found: ID does not exist" containerID="be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8" Nov 01 21:02:44.418929 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.418860 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8"} err="failed to get container status \"be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8\": rpc error: code = NotFound desc = could not find container \"be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8\": container with ID starting with be12d2dbb02544495c49ea4a7674345b01027b66ae4beec1816b45644f460be8 not found: ID does not exist" Nov 01 21:02:44.570444 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.570413 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4"] Nov 01 21:02:44.572081 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.572058 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7787bbd4d-p6kg4"] Nov 01 21:02:44.695880 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.695806 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w"] Nov 01 21:02:44.696236 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.696144 5444 cpu_manager.go:414] "containerMap: removing container" podUID="8b915478-6c59-48fe-852c-16ab3f8ccee5" containerName="payload-processing" Nov 01 21:02:44.696236 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.696156 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b915478-6c59-48fe-852c-16ab3f8ccee5" containerName="payload-processing" Nov 01 21:02:44.696236 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.696222 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b915478-6c59-48fe-852c-16ab3f8ccee5" containerName="payload-processing" Nov 01 21:02:44.700632 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.700613 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.715932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.715910 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm"] Nov 01 21:02:44.718884 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.718868 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.791310 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.791274 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w"] Nov 01 21:02:44.803940 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.803915 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm"] Nov 01 21:02:44.852364 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852323 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.852364 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852370 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.852620 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852391 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.852620 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852410 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.852620 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852527 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.852620 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852571 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.852620 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852598 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.852837 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852749 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqf2g\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.852837 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852786 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.852837 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852830 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.852995 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852873 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.852995 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.852903 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77fz\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.853074 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.853017 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.853074 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.853041 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.853074 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.853059 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.853192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.853120 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.853192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.853162 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.853192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.853183 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954344 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954258 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954344 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954310 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77fz\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954581 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954364 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954581 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954395 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954581 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954423 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954581 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954563 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954608 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954650 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954703 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954727 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954755 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954756 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954806 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.954843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954843 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.954767 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955059 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955098 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955125 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955179 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqf2g\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955206 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955230 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955315 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955331 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955352 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955458 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955588 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.955689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955614 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.955914 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.955833 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.957444 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.957411 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:44.957566 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.957501 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.957867 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.957850 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:44.957949 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:44.957883 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:45.011532 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.011501 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77fz\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:45.011998 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.011975 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-s7mnm\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:45.011998 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.011991 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqf2g\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:45.012167 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.012143 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vl74w\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:45.013985 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.013967 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:45.030971 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.030951 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:45.156972 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.156933 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b915478-6c59-48fe-852c-16ab3f8ccee5" path="/var/lib/kubelet/pods/8b915478-6c59-48fe-852c-16ab3f8ccee5/volumes" Nov 01 21:02:45.440726 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.440690 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w"] Nov 01 21:02:45.443318 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.443292 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm"] Nov 01 21:02:45.446076 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:02:45.445347 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091274e9_7255_4eae_8b15_a6c56c7e85f8.slice/crio-be418e6d5fd07b7deb6f8d6ec4ef07741e9b16d5beaf48361e19a57077d6c811 WatchSource:0}: Error finding container be418e6d5fd07b7deb6f8d6ec4ef07741e9b16d5beaf48361e19a57077d6c811: Status 404 returned error can't find the container with id be418e6d5fd07b7deb6f8d6ec4ef07741e9b16d5beaf48361e19a57077d6c811 Nov 01 21:02:45.446805 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:02:45.446777 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9a283d_3aa5_42f0_8450_efc4f5271c22.slice/crio-61cce991b5b5a882421e6fa390183fcda65e7d1963fee40731352d822eb522af WatchSource:0}: Error finding container 61cce991b5b5a882421e6fa390183fcda65e7d1963fee40731352d822eb522af: Status 404 returned error can't find the container with id 61cce991b5b5a882421e6fa390183fcda65e7d1963fee40731352d822eb522af Nov 01 21:02:45.448077 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.448041 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:45.448170 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.448119 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:45.448269 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.448165 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:45.449019 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.448983 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:45.449085 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.449067 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:45.449131 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:45.449116 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:02:46.421270 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:46.421214 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" event={"ID":"091274e9-7255-4eae-8b15-a6c56c7e85f8","Type":"ContainerStarted","Data":"fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022"} Nov 01 21:02:46.421843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:46.421328 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" event={"ID":"091274e9-7255-4eae-8b15-a6c56c7e85f8","Type":"ContainerStarted","Data":"be418e6d5fd07b7deb6f8d6ec4ef07741e9b16d5beaf48361e19a57077d6c811"} Nov 01 21:02:46.422882 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:46.422858 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" event={"ID":"1d9a283d-3aa5-42f0-8450-efc4f5271c22","Type":"ContainerStarted","Data":"c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e"} Nov 01 21:02:46.423002 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:46.422896 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" event={"ID":"1d9a283d-3aa5-42f0-8450-efc4f5271c22","Type":"ContainerStarted","Data":"61cce991b5b5a882421e6fa390183fcda65e7d1963fee40731352d822eb522af"} Nov 01 21:02:46.589661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:46.589220 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podStartSLOduration=2.589201998 podStartE2EDuration="2.589201998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:46.588139937 +0000 UTC m=+3567.532977799" watchObservedRunningTime="2026-11-01 21:02:46.589201998 +0000 UTC m=+3567.534039867" Nov 01 21:02:47.018412 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:47.018192 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:47.018412 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:47.018347 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:47.018412 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:47.018395 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:47.031941 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:47.031911 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:47.032797 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:47.032768 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:47.032912 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:47.032822 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:48.014991 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:48.014947 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:48.015498 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:48.015008 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:48.031846 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:48.031809 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:48.032013 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:48.031864 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:49.018020 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:49.017830 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:49.018020 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:49.017932 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:49.032145 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:49.032118 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:49.032261 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:49.032179 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:50.015250 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:50.015216 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:50.015426 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:50.015273 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:50.031622 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:50.031542 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:50.031622 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:50.031585 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:51.015290 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:51.015245 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:51.015473 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:51.015311 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:51.031319 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:51.031288 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:51.031438 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:51.031330 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:52.015357 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:52.015318 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:52.015994 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:52.015959 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:52.032431 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:52.032398 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:52.032560 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:52.032459 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:53.014993 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:53.014951 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 21:02:53.015176 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:53.015021 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 21:02:53.031414 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:53.031387 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-s7mnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 21:02:53.031740 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:53.031439 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 21:02:54.958407 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:54.958370 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:54.959252 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:54.959173 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:54.963239 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:54.963211 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:02:55.056461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:55.055806 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vl74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:02:55.056461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:55.055874 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:02:55.115540 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:55.115507 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:55.115992 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:55.115962 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:55.116905 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:55.116858 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:02:55.171482 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:02:55.171417 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podStartSLOduration=11.171401034 podStartE2EDuration="11.171401034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:02:46.710329248 +0000 UTC m=+3567.655167107" watchObservedRunningTime="2026-11-01 21:02:55.171401034 +0000 UTC m=+3576.116238902" Nov 01 21:03:00.034732 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:03:00.034692 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:03:23.597088 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.597050 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j"] Nov 01 21:03:23.601671 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.601654 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.656868 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.656837 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 21:03:23.700971 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.700930 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.702466 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.701467 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.705120 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.704399 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqph\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.705120 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.705082 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.709152 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.706579 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 21:03:23.709152 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.708155 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j"] Nov 01 21:03:23.709152 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.708261 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-2c97da2167ae\"" Nov 01 21:03:23.709152 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.708368 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 21:03:23.709152 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.709047 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-mkkw4\"" Nov 01 21:03:23.709152 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.709115 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 21:03:23.806053 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.806015 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.806219 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.806184 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.806284 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.806220 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqph\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.806284 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.806263 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.806792 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.806769 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.806892 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.806876 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.808651 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.808613 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.862228 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.862171 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqph\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:23.913898 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:23.913876 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:24.287198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:24.287170 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j"] Nov 01 21:03:24.290421 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:03:24.290391 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20cfe81_a811_4dc4_b9fe_7251da7c151a.slice/crio-7cb619fe788abd41d2f8695fe6e21528883254dbfa3e1773b0b79555a615fc77 WatchSource:0}: Error finding container 7cb619fe788abd41d2f8695fe6e21528883254dbfa3e1773b0b79555a615fc77: Status 404 returned error can't find the container with id 7cb619fe788abd41d2f8695fe6e21528883254dbfa3e1773b0b79555a615fc77 Nov 01 21:03:24.586712 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:24.586620 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" event={"ID":"e20cfe81-a811-4dc4-b9fe-7251da7c151a","Type":"ContainerStarted","Data":"4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f"} Nov 01 21:03:24.586712 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:24.586693 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" event={"ID":"e20cfe81-a811-4dc4-b9fe-7251da7c151a","Type":"ContainerStarted","Data":"7cb619fe788abd41d2f8695fe6e21528883254dbfa3e1773b0b79555a615fc77"} Nov 01 21:03:24.586915 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:24.586719 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:24.699779 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:24.699735 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" podStartSLOduration=1.6997212099999999 podStartE2EDuration="1.69972121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:03:24.698964622 +0000 UTC m=+3605.643802486" watchObservedRunningTime="2026-11-01 21:03:24.69972121 +0000 UTC m=+3605.644559077" Nov 01 21:03:30.035195 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:03:30.035155 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:03:35.593534 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:35.593502 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:03:57.052952 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:03:57.052916 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j"] Nov 01 21:04:00.034453 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:04:00.034416 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:04:02.053400 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.053334 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" podUID="e20cfe81-a811-4dc4-b9fe-7251da7c151a" containerName="payload-processing" containerID="cri-o://4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f" gracePeriod=25 Nov 01 21:04:02.285978 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.285951 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:04:02.417938 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.417857 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials\") pod \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " Nov 01 21:04:02.417938 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.417925 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume\") pod \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " Nov 01 21:04:02.418167 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.417986 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph\") pod \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " Nov 01 21:04:02.418167 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.418008 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume\") pod \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\" (UID: \"e20cfe81-a811-4dc4-b9fe-7251da7c151a\") " Nov 01 21:04:02.418336 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.418250 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume" pod "e20cfe81-a811-4dc4-b9fe-7251da7c151a" (UID: "e20cfe81-a811-4dc4-b9fe-7251da7c151a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:02.418434 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.418416 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume" pod "e20cfe81-a811-4dc4-b9fe-7251da7c151a" (UID: "e20cfe81-a811-4dc4-b9fe-7251da7c151a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:04:02.419924 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.419902 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials" pod "e20cfe81-a811-4dc4-b9fe-7251da7c151a" (UID: "e20cfe81-a811-4dc4-b9fe-7251da7c151a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:02.420008 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.419976 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph" pod "e20cfe81-a811-4dc4-b9fe-7251da7c151a" (UID: "e20cfe81-a811-4dc4-b9fe-7251da7c151a"). InnerVolumeSpecName "kube-api-access-2pqph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:04:02.519351 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.519316 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pqph\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-kube-api-access-2pqph\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:02.519351 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.519344 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:02.519351 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.519353 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e20cfe81-a811-4dc4-b9fe-7251da7c151a-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:02.519599 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.519362 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e20cfe81-a811-4dc4-b9fe-7251da7c151a-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:04:02.738555 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.738464 5444 generic.go:356] "Generic (PLEG): container finished" podID="e20cfe81-a811-4dc4-b9fe-7251da7c151a" containerID="4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f" exitCode=0 Nov 01 21:04:02.738555 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.738511 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" Nov 01 21:04:02.738555 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.738547 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" event={"ID":"e20cfe81-a811-4dc4-b9fe-7251da7c151a","Type":"ContainerDied","Data":"4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f"} Nov 01 21:04:02.738866 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.738595 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j" event={"ID":"e20cfe81-a811-4dc4-b9fe-7251da7c151a","Type":"ContainerDied","Data":"7cb619fe788abd41d2f8695fe6e21528883254dbfa3e1773b0b79555a615fc77"} Nov 01 21:04:02.738866 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.738616 5444 scope.go:122] "RemoveContainer" containerID="4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f" Nov 01 21:04:02.747764 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.747745 5444 scope.go:122] "RemoveContainer" containerID="4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f" Nov 01 21:04:02.748022 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:04:02.748001 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f\": container with ID starting with 4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f not found: ID does not exist" containerID="4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f" Nov 01 21:04:02.748068 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.748030 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f"} err="failed to get container status \"4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f\": rpc error: code = NotFound desc = could not find container \"4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f\": container with ID starting with 4d05b321ba7009eedb7ebc4c87d6c9c38231fc9c0f5c3c4ab8b20681b93bcc5f not found: ID does not exist" Nov 01 21:04:02.898631 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.898570 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j"] Nov 01 21:04:02.901695 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:02.901669 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7d66hg75j"] Nov 01 21:04:03.156575 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:03.156545 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20cfe81-a811-4dc4-b9fe-7251da7c151a" path="/var/lib/kubelet/pods/e20cfe81-a811-4dc4-b9fe-7251da7c151a/volumes" Nov 01 21:04:24.911008 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.910975 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt"] Nov 01 21:04:24.911419 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.911290 5444 cpu_manager.go:414] "containerMap: removing container" podUID="e20cfe81-a811-4dc4-b9fe-7251da7c151a" containerName="payload-processing" Nov 01 21:04:24.911419 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.911302 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20cfe81-a811-4dc4-b9fe-7251da7c151a" containerName="payload-processing" Nov 01 21:04:24.911419 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.911360 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20cfe81-a811-4dc4-b9fe-7251da7c151a" containerName="payload-processing" Nov 01 21:04:24.915328 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.915311 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:24.929817 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.929790 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:24.929939 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.929846 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:24.929983 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.929944 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9khw\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:24.929983 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.929975 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:24.967220 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:24.967195 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-a144a099ec31\"" Nov 01 21:04:25.016990 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.016969 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 21:04:25.016990 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.016981 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 21:04:25.017169 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.016975 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 21:04:25.017169 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.016994 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 21:04:25.017849 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.017832 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt"] Nov 01 21:04:25.030484 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.030462 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.030587 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.030501 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.030587 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.030519 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9khw\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.030587 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.030546 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.031157 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.031134 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.031277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.031254 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.032888 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.032865 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.066799 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.066777 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-zshnb\"" Nov 01 21:04:25.086461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.086440 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9khw\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.227503 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.227414 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.610112 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.610017 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt"] Nov 01 21:04:25.613612 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:04:25.613586 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b8c638_736b_449d_b8e6_daf0bb97336d.slice/crio-d69cdb04be17da177b070b48345786e09f6453f8566488bf56fc7c4b7f091f72 WatchSource:0}: Error finding container d69cdb04be17da177b070b48345786e09f6453f8566488bf56fc7c4b7f091f72: Status 404 returned error can't find the container with id d69cdb04be17da177b070b48345786e09f6453f8566488bf56fc7c4b7f091f72 Nov 01 21:04:25.841598 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.841570 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" event={"ID":"10b8c638-736b-449d-b8e6-daf0bb97336d","Type":"ContainerStarted","Data":"6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96"} Nov 01 21:04:25.841764 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.841620 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" event={"ID":"10b8c638-736b-449d-b8e6-daf0bb97336d","Type":"ContainerStarted","Data":"d69cdb04be17da177b070b48345786e09f6453f8566488bf56fc7c4b7f091f72"} Nov 01 21:04:25.841764 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.841708 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:04:25.955370 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:25.955275 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" podStartSLOduration=1.955259648 podStartE2EDuration="1.955259648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:04:25.954389142 +0000 UTC m=+3666.899227012" watchObservedRunningTime="2026-11-01 21:04:25.955259648 +0000 UTC m=+3666.900097517" Nov 01 21:04:30.033455 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:04:30.033415 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:04:36.849934 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:04:36.849907 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:05:00.034359 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:05:00.034280 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:05:30.034035 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:05:30.033998 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:05:37.807914 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.807884 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8"] Nov 01 21:05:37.811103 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.811085 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:37.869693 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.869673 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-9c12faaff243\"" Nov 01 21:05:37.890176 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.890156 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8"] Nov 01 21:05:37.930555 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.930530 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:37.930669 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.930564 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865jg\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:37.930725 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.930671 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:37.930725 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:37.930699 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.031380 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.031358 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.031470 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.031390 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865jg\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.031470 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.031427 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.031470 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.031449 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.032056 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.032033 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.032151 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.032131 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.033841 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.033823 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.087495 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.087444 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-865jg\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.122966 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.122943 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:38.489497 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:38.489426 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8"] Nov 01 21:05:38.492561 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:05:38.492536 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4413ac_0d11_4391_b47d_db9acd145914.slice/crio-6d2ffe123c4cb9a43f6d6ab0d6f2c8440ccd01f094ca7dd9acd5247a5500aadd WatchSource:0}: Error finding container 6d2ffe123c4cb9a43f6d6ab0d6f2c8440ccd01f094ca7dd9acd5247a5500aadd: Status 404 returned error can't find the container with id 6d2ffe123c4cb9a43f6d6ab0d6f2c8440ccd01f094ca7dd9acd5247a5500aadd Nov 01 21:05:39.138390 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:39.138359 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" event={"ID":"5a4413ac-0d11-4391-b47d-db9acd145914","Type":"ContainerStarted","Data":"993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4"} Nov 01 21:05:39.138815 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:39.138408 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" event={"ID":"5a4413ac-0d11-4391-b47d-db9acd145914","Type":"ContainerStarted","Data":"6d2ffe123c4cb9a43f6d6ab0d6f2c8440ccd01f094ca7dd9acd5247a5500aadd"} Nov 01 21:05:39.138815 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:39.138477 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:39.250729 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:39.250680 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" podStartSLOduration=2.250664203 podStartE2EDuration="2.250664203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:05:39.2505785 +0000 UTC m=+3740.195416370" watchObservedRunningTime="2026-11-01 21:05:39.250664203 +0000 UTC m=+3740.195502065" Nov 01 21:05:50.146210 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:50.146175 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:05:50.308213 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:50.308185 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt"] Nov 01 21:05:55.308788 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.308729 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" podUID="10b8c638-736b-449d-b8e6-daf0bb97336d" containerName="payload-processing" containerID="cri-o://6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96" gracePeriod=25 Nov 01 21:05:55.553611 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.553588 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:05:55.669079 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.669018 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume\") pod \"10b8c638-736b-449d-b8e6-daf0bb97336d\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " Nov 01 21:05:55.669079 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.669052 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume\") pod \"10b8c638-736b-449d-b8e6-daf0bb97336d\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " Nov 01 21:05:55.669242 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.669109 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw\") pod \"10b8c638-736b-449d-b8e6-daf0bb97336d\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " Nov 01 21:05:55.669242 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.669135 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials\") pod \"10b8c638-736b-449d-b8e6-daf0bb97336d\" (UID: \"10b8c638-736b-449d-b8e6-daf0bb97336d\") " Nov 01 21:05:55.669474 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.669446 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume" pod "10b8c638-736b-449d-b8e6-daf0bb97336d" (UID: "10b8c638-736b-449d-b8e6-daf0bb97336d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:55.669546 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.669488 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume" pod "10b8c638-736b-449d-b8e6-daf0bb97336d" (UID: "10b8c638-736b-449d-b8e6-daf0bb97336d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:05:55.671289 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.671264 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials" pod "10b8c638-736b-449d-b8e6-daf0bb97336d" (UID: "10b8c638-736b-449d-b8e6-daf0bb97336d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:55.671512 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.671492 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw" pod "10b8c638-736b-449d-b8e6-daf0bb97336d" (UID: "10b8c638-736b-449d-b8e6-daf0bb97336d"). InnerVolumeSpecName "kube-api-access-s9khw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:05:55.770248 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.770218 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:55.770248 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.770245 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9khw\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-kube-api-access-s9khw\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:55.770376 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.770260 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10b8c638-736b-449d-b8e6-daf0bb97336d-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:55.770376 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:55.770273 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b8c638-736b-449d-b8e6-daf0bb97336d-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:05:56.213050 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.213017 5444 generic.go:356] "Generic (PLEG): container finished" podID="10b8c638-736b-449d-b8e6-daf0bb97336d" containerID="6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96" exitCode=0 Nov 01 21:05:56.213191 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.213065 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" Nov 01 21:05:56.213191 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.213070 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" event={"ID":"10b8c638-736b-449d-b8e6-daf0bb97336d","Type":"ContainerDied","Data":"6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96"} Nov 01 21:05:56.213322 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.213219 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt" event={"ID":"10b8c638-736b-449d-b8e6-daf0bb97336d","Type":"ContainerDied","Data":"d69cdb04be17da177b070b48345786e09f6453f8566488bf56fc7c4b7f091f72"} Nov 01 21:05:56.213322 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.213264 5444 scope.go:122] "RemoveContainer" containerID="6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96" Nov 01 21:05:56.221972 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.221954 5444 scope.go:122] "RemoveContainer" containerID="6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96" Nov 01 21:05:56.222252 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:05:56.222235 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96\": container with ID starting with 6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96 not found: ID does not exist" containerID="6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96" Nov 01 21:05:56.222323 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.222258 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96"} err="failed to get container status \"6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96\": rpc error: code = NotFound desc = could not find container \"6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96\": container with ID starting with 6d0247dfa1608f4559cc486a9b1017b5aa7f1dbf388d14cbef27f924ab99ca96 not found: ID does not exist" Nov 01 21:05:56.373675 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.373630 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt"] Nov 01 21:05:56.422704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:56.422679 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-55ddnlsqt"] Nov 01 21:05:57.156944 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:05:57.156908 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b8c638-736b-449d-b8e6-daf0bb97336d" path="/var/lib/kubelet/pods/10b8c638-736b-449d-b8e6-daf0bb97336d/volumes" Nov 01 21:06:00.034236 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:06:00.034202 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:06:30.036989 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:06:30.036948 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:07:00.034951 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:07:00.034916 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:07:30.033927 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:07:30.033892 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:08:00.034308 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:08:00.034235 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:08:30.035969 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:08:30.035929 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:09:00.035008 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:09:00.034964 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:09:26.612568 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:26.612534 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8"] Nov 01 21:09:30.035581 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:09:30.035537 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:09:31.613171 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.613087 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" podUID="5a4413ac-0d11-4391-b47d-db9acd145914" containerName="payload-processing" containerID="cri-o://993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4" gracePeriod=25 Nov 01 21:09:31.869079 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.869008 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:09:31.955667 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.955609 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume\") pod \"5a4413ac-0d11-4391-b47d-db9acd145914\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " Nov 01 21:09:31.955862 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.955678 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg\") pod \"5a4413ac-0d11-4391-b47d-db9acd145914\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " Nov 01 21:09:31.955862 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.955717 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials\") pod \"5a4413ac-0d11-4391-b47d-db9acd145914\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " Nov 01 21:09:31.955862 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.955756 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume\") pod \"5a4413ac-0d11-4391-b47d-db9acd145914\" (UID: \"5a4413ac-0d11-4391-b47d-db9acd145914\") " Nov 01 21:09:31.956121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.956095 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume" pod "5a4413ac-0d11-4391-b47d-db9acd145914" (UID: "5a4413ac-0d11-4391-b47d-db9acd145914"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:31.956183 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.956162 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume" pod "5a4413ac-0d11-4391-b47d-db9acd145914" (UID: "5a4413ac-0d11-4391-b47d-db9acd145914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:31.957887 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.957863 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials" pod "5a4413ac-0d11-4391-b47d-db9acd145914" (UID: "5a4413ac-0d11-4391-b47d-db9acd145914"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:31.957967 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:31.957954 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg" pod "5a4413ac-0d11-4391-b47d-db9acd145914" (UID: "5a4413ac-0d11-4391-b47d-db9acd145914"). InnerVolumeSpecName "kube-api-access-865jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:32.057257 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.057204 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:32.057257 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.057246 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:32.057257 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.057255 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a4413ac-0d11-4391-b47d-db9acd145914-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:32.057257 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.057264 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-865jg\" (UniqueName: \"kubernetes.io/projected/5a4413ac-0d11-4391-b47d-db9acd145914-kube-api-access-865jg\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:32.095771 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.095733 5444 generic.go:356] "Generic (PLEG): container finished" podID="5a4413ac-0d11-4391-b47d-db9acd145914" containerID="993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4" exitCode=0 Nov 01 21:09:32.095974 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.095784 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" Nov 01 21:09:32.095974 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.095816 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" event={"ID":"5a4413ac-0d11-4391-b47d-db9acd145914","Type":"ContainerDied","Data":"993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4"} Nov 01 21:09:32.095974 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.095882 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8" event={"ID":"5a4413ac-0d11-4391-b47d-db9acd145914","Type":"ContainerDied","Data":"6d2ffe123c4cb9a43f6d6ab0d6f2c8440ccd01f094ca7dd9acd5247a5500aadd"} Nov 01 21:09:32.095974 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.095916 5444 scope.go:122] "RemoveContainer" containerID="993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4" Nov 01 21:09:32.104563 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.104543 5444 scope.go:122] "RemoveContainer" containerID="993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4" Nov 01 21:09:32.104902 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:09:32.104884 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4\": container with ID starting with 993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4 not found: ID does not exist" containerID="993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4" Nov 01 21:09:32.104971 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.104918 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4"} err="failed to get container status \"993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4\": rpc error: code = NotFound desc = could not find container \"993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4\": container with ID starting with 993bdc1cafbd4c39cb93935e1664d104420d5cd5070ba32bd5e6cc61b326edb4 not found: ID does not exist" Nov 01 21:09:32.259250 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.259217 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8"] Nov 01 21:09:32.262118 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:32.262094 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-77cd4gnt8"] Nov 01 21:09:33.157075 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:33.157045 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4413ac-0d11-4391-b47d-db9acd145914" path="/var/lib/kubelet/pods/5a4413ac-0d11-4391-b47d-db9acd145914/volumes" Nov 01 21:09:47.678168 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:47.678130 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm"] Nov 01 21:09:47.678580 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:47.678351 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" containerID="cri-o://c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e" gracePeriod=30 Nov 01 21:09:47.727856 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:47.727821 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w"] Nov 01 21:09:47.728103 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:47.728077 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" containerID="cri-o://fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022" gracePeriod=30 Nov 01 21:09:52.967273 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:52.967240 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:09:53.001933 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.001906 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:09:53.039995 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.039952 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040005 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040042 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040077 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040100 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040129 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040198 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040191 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040225 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040275 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.040518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040322 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs\") pod \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\" (UID: \"1d9a283d-3aa5-42f0-8450-efc4f5271c22\") " Nov 01 21:09:53.040518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040376 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.040518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040462 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.040777 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040602 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.040777 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040628 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.040777 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040632 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:53.040777 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040659 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.040777 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.040630 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.042458 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.042429 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:09:53.042564 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.042479 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.042564 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.042494 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:53.042564 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.042497 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz" pod "1d9a283d-3aa5-42f0-8450-efc4f5271c22" (UID: "1d9a283d-3aa5-42f0-8450-efc4f5271c22"). InnerVolumeSpecName "kube-api-access-d77fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:53.141231 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141196 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141231 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141237 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141269 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141291 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141325 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141349 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141375 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141450 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141834 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141491 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g\") pod \"091274e9-7255-4eae-8b15-a6c56c7e85f8\" (UID: \"091274e9-7255-4eae-8b15-a6c56c7e85f8\") " Nov 01 21:09:53.141834 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141521 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.141834 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141699 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.141834 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141805 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.141834 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141826 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141843 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141848 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141860 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141895 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9a283d-3aa5-42f0-8450-efc4f5271c22-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141911 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141925 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d77fz\" (UniqueName: \"kubernetes.io/projected/1d9a283d-3aa5-42f0-8450-efc4f5271c22-kube-api-access-d77fz\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141940 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9a283d-3aa5-42f0-8450-efc4f5271c22-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.142105 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.141994 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.142506 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.142145 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:09:53.143708 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.143684 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:09:53.143831 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.143727 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "kube-api-access-qqf2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:53.143933 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.143914 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:09:53.144004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.143949 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token" pod "091274e9-7255-4eae-8b15-a6c56c7e85f8" (UID: "091274e9-7255-4eae-8b15-a6c56c7e85f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:09:53.190837 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.190804 5444 generic.go:356] "Generic (PLEG): container finished" podID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerID="fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022" exitCode=0 Nov 01 21:09:53.191019 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.190851 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" Nov 01 21:09:53.191019 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.190884 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" event={"ID":"091274e9-7255-4eae-8b15-a6c56c7e85f8","Type":"ContainerDied","Data":"fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022"} Nov 01 21:09:53.191019 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.190933 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w" event={"ID":"091274e9-7255-4eae-8b15-a6c56c7e85f8","Type":"ContainerDied","Data":"be418e6d5fd07b7deb6f8d6ec4ef07741e9b16d5beaf48361e19a57077d6c811"} Nov 01 21:09:53.191019 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.190961 5444 scope.go:122] "RemoveContainer" containerID="fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022" Nov 01 21:09:53.192150 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.192125 5444 generic.go:356] "Generic (PLEG): container finished" podID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerID="c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e" exitCode=0 Nov 01 21:09:53.192263 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.192153 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" event={"ID":"1d9a283d-3aa5-42f0-8450-efc4f5271c22","Type":"ContainerDied","Data":"c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e"} Nov 01 21:09:53.192263 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.192175 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" Nov 01 21:09:53.192263 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.192193 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm" event={"ID":"1d9a283d-3aa5-42f0-8450-efc4f5271c22","Type":"ContainerDied","Data":"61cce991b5b5a882421e6fa390183fcda65e7d1963fee40731352d822eb522af"} Nov 01 21:09:53.199539 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.199521 5444 scope.go:122] "RemoveContainer" containerID="fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022" Nov 01 21:09:53.199851 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:09:53.199827 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022\": container with ID starting with fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022 not found: ID does not exist" containerID="fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022" Nov 01 21:09:53.199899 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.199860 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022"} err="failed to get container status \"fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022\": rpc error: code = NotFound desc = could not find container \"fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022\": container with ID starting with fe212e49dd257241e1bdcaf1735c9dd01d9fae9b75b48f22f69a77c8d881d022 not found: ID does not exist" Nov 01 21:09:53.199899 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.199876 5444 scope.go:122] "RemoveContainer" containerID="c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e" Nov 01 21:09:53.207677 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.207653 5444 scope.go:122] "RemoveContainer" containerID="c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e" Nov 01 21:09:53.207965 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:09:53.207945 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e\": container with ID starting with c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e not found: ID does not exist" containerID="c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e" Nov 01 21:09:53.208032 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.207979 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e"} err="failed to get container status \"c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e\": rpc error: code = NotFound desc = could not find container \"c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e\": container with ID starting with c48ea0b2741e0d43b095c1912552d1655ce94fac6d32b796dabfb74ed20cb67e not found: ID does not exist" Nov 01 21:09:53.243301 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243251 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.243301 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243293 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.243301 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243304 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/091274e9-7255-4eae-8b15-a6c56c7e85f8-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.243301 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243315 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.243569 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243328 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/091274e9-7255-4eae-8b15-a6c56c7e85f8-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.243569 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243339 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqf2g\" (UniqueName: \"kubernetes.io/projected/091274e9-7255-4eae-8b15-a6c56c7e85f8-kube-api-access-qqf2g\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.243569 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.243348 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/091274e9-7255-4eae-8b15-a6c56c7e85f8-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:09:53.354882 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.354843 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm"] Nov 01 21:09:53.360651 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.360610 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-s7mnm"] Nov 01 21:09:53.468411 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.468325 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w"] Nov 01 21:09:53.473797 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:53.473774 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vl74w"] Nov 01 21:09:55.157563 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:55.157522 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" path="/var/lib/kubelet/pods/091274e9-7255-4eae-8b15-a6c56c7e85f8/volumes" Nov 01 21:09:55.158108 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:09:55.158087 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" path="/var/lib/kubelet/pods/1d9a283d-3aa5-42f0-8450-efc4f5271c22/volumes" Nov 01 21:10:00.034748 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:10:00.034711 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:10:30.034492 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:10:30.034456 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:11:00.034825 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:11:00.034742 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:11:30.037406 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:11:30.037358 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:12:00.033858 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:12:00.033817 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:12:30.034568 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:12:30.034532 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:13:00.035374 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:13:00.035340 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:13:30.039935 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:13:30.039817 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:14:00.033969 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:14:00.033883 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:14:04.912424 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912390 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2"] Nov 01 21:14:04.912936 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912864 5444 cpu_manager.go:414] "containerMap: removing container" podUID="5a4413ac-0d11-4391-b47d-db9acd145914" containerName="payload-processing" Nov 01 21:14:04.912936 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912881 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4413ac-0d11-4391-b47d-db9acd145914" containerName="payload-processing" Nov 01 21:14:04.912936 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912906 5444 cpu_manager.go:414] "containerMap: removing container" podUID="10b8c638-736b-449d-b8e6-daf0bb97336d" containerName="payload-processing" Nov 01 21:14:04.912936 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912913 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b8c638-736b-449d-b8e6-daf0bb97336d" containerName="payload-processing" Nov 01 21:14:04.913066 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912952 5444 cpu_manager.go:414] "containerMap: removing container" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" Nov 01 21:14:04.913066 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912961 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" Nov 01 21:14:04.913066 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912974 5444 cpu_manager.go:414] "containerMap: removing container" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" Nov 01 21:14:04.913066 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.912981 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" Nov 01 21:14:04.913066 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.913046 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9a283d-3aa5-42f0-8450-efc4f5271c22" containerName="istio-proxy" Nov 01 21:14:04.913066 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.913056 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4413ac-0d11-4391-b47d-db9acd145914" containerName="payload-processing" Nov 01 21:14:04.913248 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.913067 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b8c638-736b-449d-b8e6-daf0bb97336d" containerName="payload-processing" Nov 01 21:14:04.913248 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.913080 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="091274e9-7255-4eae-8b15-a6c56c7e85f8" containerName="istio-proxy" Nov 01 21:14:04.916285 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.916259 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:04.968579 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.968537 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-a144a099ec31\"" Nov 01 21:14:04.968579 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.968561 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-f1b0d0f19d3d\"" Nov 01 21:14:04.968579 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:04.968544 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"agc-cred-9c12faaff243\"" Nov 01 21:14:05.017954 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.017919 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"routing-overlay\"" Nov 01 21:14:05.018162 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.018012 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"kube-root-ca.crt\"" Nov 01 21:14:05.018162 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.018013 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"openshift-service-ca.crt\"" Nov 01 21:14:05.018162 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.018087 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-e2e--a68bca68-dockercfg-zshnb\"" Nov 01 21:14:05.018162 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.018142 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-99d504\"/\"payload-processing-external-model-plugins-e2e-extmod-a-99d504\"" Nov 01 21:14:05.019018 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.018996 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2"] Nov 01 21:14:05.031536 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.031506 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.031704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.031562 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9kn\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.031704 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.031628 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.031799 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.031732 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.132831 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.132785 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9kn\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.132831 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.132833 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.133061 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.132956 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.133061 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.133020 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.133628 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.133601 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.133784 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.133734 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.135647 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.135609 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.189341 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.189255 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9kn\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn\") pod \"payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.228982 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.228949 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:05.599412 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:05.599371 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2"] Nov 01 21:14:05.610559 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:14:05.610528 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf789929_7751_458f_85fd_232e7109a081.slice/crio-a2cfbc8d04b2f4748dca7cde5c6a409bf2cb428f88636b775989727166c14cf9 WatchSource:0}: Error finding container a2cfbc8d04b2f4748dca7cde5c6a409bf2cb428f88636b775989727166c14cf9: Status 404 returned error can't find the container with id a2cfbc8d04b2f4748dca7cde5c6a409bf2cb428f88636b775989727166c14cf9 Nov 01 21:14:06.176817 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:06.176781 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" event={"ID":"cf789929-7751-458f-85fd-232e7109a081","Type":"ContainerStarted","Data":"9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f"} Nov 01 21:14:06.177262 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:06.176841 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" event={"ID":"cf789929-7751-458f-85fd-232e7109a081","Type":"ContainerStarted","Data":"a2cfbc8d04b2f4748dca7cde5c6a409bf2cb428f88636b775989727166c14cf9"} Nov 01 21:14:06.177262 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:06.176931 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:06.290550 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:06.290488 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" podStartSLOduration=2.290468922 podStartE2EDuration="2.290468922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:06.289183388 +0000 UTC m=+4247.234021256" watchObservedRunningTime="2026-11-01 21:14:06.290468922 +0000 UTC m=+4247.235306793" Nov 01 21:14:09.880845 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:09.880811 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2"] Nov 01 21:14:12.384548 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.384517 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr"] Nov 01 21:14:12.387999 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.387976 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.440143 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.440109 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-14958f7d1994\"" Nov 01 21:14:12.440143 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.440125 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"routing-overlay\"" Nov 01 21:14:12.489836 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.489791 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6a675a\"" Nov 01 21:14:12.489836 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.489812 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"kube-root-ca.crt\"" Nov 01 21:14:12.489836 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.489810 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"openshift-service-ca.crt\"" Nov 01 21:14:12.490144 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.489803 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"agc-cred-dd468d87ff44\"" Nov 01 21:14:12.490339 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.490317 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6a675a\"/\"payload-processing-external-model-e2e--a80640ba-dockercfg-nnqcq\"" Nov 01 21:14:12.492337 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.492310 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr"] Nov 01 21:14:12.498722 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.498661 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.498905 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.498788 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.498905 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.498861 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6fj\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.498905 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.498897 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.600281 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.600235 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.600487 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.600294 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.600487 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.600320 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6fj\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.600487 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.600338 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.601030 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.601004 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.601135 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.601107 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.603073 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.603050 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.656056 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.655970 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6fj\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:12.701526 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:12.701494 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:13.070208 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:13.070173 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr"] Nov 01 21:14:13.073732 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:14:13.073703 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d904c42_164a_449e_a9a8_715b3090497d.slice/crio-687154de437f094fc41fb746a0ca0dd5d6d6dab3cb194503be265568cfaac735 WatchSource:0}: Error finding container 687154de437f094fc41fb746a0ca0dd5d6d6dab3cb194503be265568cfaac735: Status 404 returned error can't find the container with id 687154de437f094fc41fb746a0ca0dd5d6d6dab3cb194503be265568cfaac735 Nov 01 21:14:13.204796 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:13.204759 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" event={"ID":"4d904c42-164a-449e-a9a8-715b3090497d","Type":"ContainerStarted","Data":"397a88a64ee68012a8c01c0a8f82404fae7c7cd8087aca540809f2c3f7f61ba7"} Nov 01 21:14:13.204992 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:13.204815 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" event={"ID":"4d904c42-164a-449e-a9a8-715b3090497d","Type":"ContainerStarted","Data":"687154de437f094fc41fb746a0ca0dd5d6d6dab3cb194503be265568cfaac735"} Nov 01 21:14:13.204992 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:13.204921 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:13.318211 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:13.318145 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" podStartSLOduration=1.318128745 podStartE2EDuration="1.318128745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:13.317443695 +0000 UTC m=+4254.262281563" watchObservedRunningTime="2026-11-01 21:14:13.318128745 +0000 UTC m=+4254.262966613" Nov 01 21:14:14.881302 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:14.881253 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" podUID="cf789929-7751-458f-85fd-232e7109a081" containerName="payload-processing" containerID="cri-o://9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f" gracePeriod=25 Nov 01 21:14:15.123854 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.123828 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:15.213539 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.213451 5444 generic.go:356] "Generic (PLEG): container finished" podID="cf789929-7751-458f-85fd-232e7109a081" containerID="9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f" exitCode=0 Nov 01 21:14:15.213539 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.213503 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" Nov 01 21:14:15.213539 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.213528 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" event={"ID":"cf789929-7751-458f-85fd-232e7109a081","Type":"ContainerDied","Data":"9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f"} Nov 01 21:14:15.213827 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.213578 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2" event={"ID":"cf789929-7751-458f-85fd-232e7109a081","Type":"ContainerDied","Data":"a2cfbc8d04b2f4748dca7cde5c6a409bf2cb428f88636b775989727166c14cf9"} Nov 01 21:14:15.213827 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.213606 5444 scope.go:122] "RemoveContainer" containerID="9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f" Nov 01 21:14:15.220885 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.220860 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume\") pod \"cf789929-7751-458f-85fd-232e7109a081\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " Nov 01 21:14:15.220989 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.220896 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume\") pod \"cf789929-7751-458f-85fd-232e7109a081\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " Nov 01 21:14:15.220989 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.220925 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn\") pod \"cf789929-7751-458f-85fd-232e7109a081\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " Nov 01 21:14:15.220989 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.220957 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials\") pod \"cf789929-7751-458f-85fd-232e7109a081\" (UID: \"cf789929-7751-458f-85fd-232e7109a081\") " Nov 01 21:14:15.221219 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.221195 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume" pod "cf789929-7751-458f-85fd-232e7109a081" (UID: "cf789929-7751-458f-85fd-232e7109a081"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:15.221594 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.221470 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume" pod "cf789929-7751-458f-85fd-232e7109a081" (UID: "cf789929-7751-458f-85fd-232e7109a081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:15.221689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.221656 5444 scope.go:122] "RemoveContainer" containerID="9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f" Nov 01 21:14:15.221972 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:14:15.221950 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f\": container with ID starting with 9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f not found: ID does not exist" containerID="9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f" Nov 01 21:14:15.222068 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.222039 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f"} err="failed to get container status \"9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f\": rpc error: code = NotFound desc = could not find container \"9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f\": container with ID starting with 9235b315411a8043b19cb82c4fc8f680597ef7fd03fb6f179ff610623846ce6f not found: ID does not exist" Nov 01 21:14:15.223135 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.223109 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials" pod "cf789929-7751-458f-85fd-232e7109a081" (UID: "cf789929-7751-458f-85fd-232e7109a081"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:15.223220 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.223115 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn" pod "cf789929-7751-458f-85fd-232e7109a081" (UID: "cf789929-7751-458f-85fd-232e7109a081"). InnerVolumeSpecName "kube-api-access-kq9kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:15.322192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.322153 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:15.322192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.322187 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:15.322192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.322200 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf789929-7751-458f-85fd-232e7109a081-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:15.322192 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.322208 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9kn\" (UniqueName: \"kubernetes.io/projected/cf789929-7751-458f-85fd-232e7109a081-kube-api-access-kq9kn\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:15.631439 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.631399 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2"] Nov 01 21:14:15.680709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:15.680671 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-99d504/payload-processing-external-model-e2e-extmod-a-99d504-c6b5m7hk2"] Nov 01 21:14:17.158382 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:17.158342 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf789929-7751-458f-85fd-232e7109a081" path="/var/lib/kubelet/pods/cf789929-7751-458f-85fd-232e7109a081/volumes" Nov 01 21:14:24.211803 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:24.211762 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:30.034028 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:14:30.033994 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:14:34.210371 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:34.210326 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:38.096766 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.096731 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr"] Nov 01 21:14:38.098063 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.098032 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:14:38.124547 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.124509 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg"] Nov 01 21:14:38.124868 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.124852 5444 cpu_manager.go:414] "containerMap: removing container" podUID="cf789929-7751-458f-85fd-232e7109a081" containerName="payload-processing" Nov 01 21:14:38.124922 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.124870 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf789929-7751-458f-85fd-232e7109a081" containerName="payload-processing" Nov 01 21:14:38.124958 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.124949 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf789929-7751-458f-85fd-232e7109a081" containerName="payload-processing" Nov 01 21:14:38.129059 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.129034 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.184116 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.184078 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg"] Nov 01 21:14:38.200741 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.200694 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzcjv\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.200919 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.200813 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.200919 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.200873 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.200919 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.200902 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.301764 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.301723 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.301764 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.301772 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.302054 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.301846 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzcjv\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.302054 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.301888 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.302546 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.302521 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.302666 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.302556 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.304170 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.304150 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.357854 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.357773 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzcjv\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv\") pod \"payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.441918 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.441874 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:38.821073 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:38.821041 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg"] Nov 01 21:14:38.823569 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:14:38.823521 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1551d5_5eee_4e58_a380_93c67298b86b.slice/crio-aba61f33c2cabcecd55f256e343d8fa43db2db0f8bd87ce3a0bd06b22ffac9e6 WatchSource:0}: Error finding container aba61f33c2cabcecd55f256e343d8fa43db2db0f8bd87ce3a0bd06b22ffac9e6: Status 404 returned error can't find the container with id aba61f33c2cabcecd55f256e343d8fa43db2db0f8bd87ce3a0bd06b22ffac9e6 Nov 01 21:14:39.314072 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:39.314037 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" event={"ID":"fa1551d5-5eee-4e58-a380-93c67298b86b","Type":"ContainerStarted","Data":"076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9"} Nov 01 21:14:39.314475 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:39.314086 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" event={"ID":"fa1551d5-5eee-4e58-a380-93c67298b86b","Type":"ContainerStarted","Data":"aba61f33c2cabcecd55f256e343d8fa43db2db0f8bd87ce3a0bd06b22ffac9e6"} Nov 01 21:14:39.314475 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:39.314173 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:14:39.426480 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:39.426399 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" podStartSLOduration=1.4263807210000001 podStartE2EDuration="1.426380721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:14:39.425184664 +0000 UTC m=+4280.370022533" watchObservedRunningTime="2026-11-01 21:14:39.426380721 +0000 UTC m=+4280.371218589" Nov 01 21:14:43.097714 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.097626 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" containerID="cri-o://397a88a64ee68012a8c01c0a8f82404fae7c7cd8087aca540809f2c3f7f61ba7" gracePeriod=25 Nov 01 21:14:43.336079 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.336051 5444 generic.go:356] "Generic (PLEG): container finished" podID="4d904c42-164a-449e-a9a8-715b3090497d" containerID="397a88a64ee68012a8c01c0a8f82404fae7c7cd8087aca540809f2c3f7f61ba7" exitCode=0 Nov 01 21:14:43.336197 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.336144 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" event={"ID":"4d904c42-164a-449e-a9a8-715b3090497d","Type":"ContainerDied","Data":"397a88a64ee68012a8c01c0a8f82404fae7c7cd8087aca540809f2c3f7f61ba7"} Nov 01 21:14:43.336251 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.336197 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" event={"ID":"4d904c42-164a-449e-a9a8-715b3090497d","Type":"ContainerDied","Data":"687154de437f094fc41fb746a0ca0dd5d6d6dab3cb194503be265568cfaac735"} Nov 01 21:14:43.336251 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.336215 5444 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="687154de437f094fc41fb746a0ca0dd5d6d6dab3cb194503be265568cfaac735" Nov 01 21:14:43.337006 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.336993 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:43.443520 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.443439 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume\") pod \"4d904c42-164a-449e-a9a8-715b3090497d\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " Nov 01 21:14:43.443520 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.443493 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials\") pod \"4d904c42-164a-449e-a9a8-715b3090497d\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " Nov 01 21:14:43.443520 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.443522 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj\") pod \"4d904c42-164a-449e-a9a8-715b3090497d\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " Nov 01 21:14:43.443848 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.443551 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume\") pod \"4d904c42-164a-449e-a9a8-715b3090497d\" (UID: \"4d904c42-164a-449e-a9a8-715b3090497d\") " Nov 01 21:14:43.443988 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.443963 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume" pod "4d904c42-164a-449e-a9a8-715b3090497d" (UID: "4d904c42-164a-449e-a9a8-715b3090497d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:43.444069 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.444041 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume" pod "4d904c42-164a-449e-a9a8-715b3090497d" (UID: "4d904c42-164a-449e-a9a8-715b3090497d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:14:43.445683 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.445660 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj" pod "4d904c42-164a-449e-a9a8-715b3090497d" (UID: "4d904c42-164a-449e-a9a8-715b3090497d"). InnerVolumeSpecName "kube-api-access-xv6fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:43.445747 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.445678 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials" pod "4d904c42-164a-449e-a9a8-715b3090497d" (UID: "4d904c42-164a-449e-a9a8-715b3090497d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:14:43.545092 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.545056 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6fj\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-kube-api-access-xv6fj\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:43.545092 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.545085 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:43.545092 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.545095 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d904c42-164a-449e-a9a8-715b3090497d-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:43.545330 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:43.545104 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d904c42-164a-449e-a9a8-715b3090497d-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:14:44.339573 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:44.339541 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr" Nov 01 21:14:44.499957 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:44.499921 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr"] Nov 01 21:14:44.549380 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:44.549351 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-5d75hjznr"] Nov 01 21:14:45.157909 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:45.157876 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d904c42-164a-449e-a9a8-715b3090497d" path="/var/lib/kubelet/pods/4d904c42-164a-449e-a9a8-715b3090497d/volumes" Nov 01 21:14:50.326657 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:14:50.326597 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:15:00.035295 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:15:00.035254 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:15:00.165565 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.165531 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7"] Nov 01 21:15:00.166001 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.165982 5444 cpu_manager.go:414] "containerMap: removing container" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" Nov 01 21:15:00.166001 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.166001 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" Nov 01 21:15:00.166318 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.166089 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d904c42-164a-449e-a9a8-715b3090497d" containerName="payload-processing" Nov 01 21:15:00.169058 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.169043 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.227546 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.227510 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7"] Nov 01 21:15:00.272986 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.272953 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tr6mg\"" Nov 01 21:15:00.291925 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.291853 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892795-mjbq7\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.291925 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.291916 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jts\" (UniqueName: \"kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts\") pod \"maas-api-key-cleanup-29892795-mjbq7\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.325093 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.325056 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:15:00.392388 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.392356 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jts\" (UniqueName: \"kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts\") pod \"maas-api-key-cleanup-29892795-mjbq7\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.392547 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.392426 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892795-mjbq7\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.393049 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.393029 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892795-mjbq7\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.448216 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.448181 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jts\" (UniqueName: \"kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts\") pod \"maas-api-key-cleanup-29892795-mjbq7\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.482039 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.482003 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:00.855877 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:00.855847 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7"] Nov 01 21:15:00.859299 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:15:00.859272 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95092934_7ae3_480b_8d85_f0882e449737.slice/crio-0cb3c77e4ef35547d2867a2993c7217e80cac8904b7b3e769055c08d7bc8ad7b WatchSource:0}: Error finding container 0cb3c77e4ef35547d2867a2993c7217e80cac8904b7b3e769055c08d7bc8ad7b: Status 404 returned error can't find the container with id 0cb3c77e4ef35547d2867a2993c7217e80cac8904b7b3e769055c08d7bc8ad7b Nov 01 21:15:01.408754 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:01.408721 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerStarted","Data":"c497ce0cb0e576719a1b24f33048eb7288684c8073fe94f8ea72e27789bb9827"} Nov 01 21:15:01.409163 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:01.408771 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerStarted","Data":"0cb3c77e4ef35547d2867a2993c7217e80cac8904b7b3e769055c08d7bc8ad7b"} Nov 01 21:15:01.520136 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:01.520084 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" podStartSLOduration=1.5200698670000001 podStartE2EDuration="1.520069867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:15:01.519040044 +0000 UTC m=+4302.463877912" watchObservedRunningTime="2026-11-01 21:15:01.520069867 +0000 UTC m=+4302.464907725" Nov 01 21:15:02.414753 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:02.414714 5444 generic.go:356] "Generic (PLEG): container finished" podID="95092934-7ae3-480b-8d85-f0882e449737" containerID="c497ce0cb0e576719a1b24f33048eb7288684c8073fe94f8ea72e27789bb9827" exitCode=60 Nov 01 21:15:02.415156 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:02.414808 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerDied","Data":"c497ce0cb0e576719a1b24f33048eb7288684c8073fe94f8ea72e27789bb9827"} Nov 01 21:15:02.415156 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:02.415020 5444 scope.go:122] "RemoveContainer" containerID="c497ce0cb0e576719a1b24f33048eb7288684c8073fe94f8ea72e27789bb9827" Nov 01 21:15:02.608789 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:02.608757 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg"] Nov 01 21:15:02.610438 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:02.610406 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 21:15:03.421218 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:03.421127 5444 generic.go:356] "Generic (PLEG): container finished" podID="95092934-7ae3-480b-8d85-f0882e449737" containerID="8381705b6880ece47490069245d953ad11f55c9a8931f795e68cb13b82aba720" exitCode=60 Nov 01 21:15:03.421218 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:03.421205 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerDied","Data":"8381705b6880ece47490069245d953ad11f55c9a8931f795e68cb13b82aba720"} Nov 01 21:15:03.421785 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:03.421260 5444 scope.go:122] "RemoveContainer" containerID="c497ce0cb0e576719a1b24f33048eb7288684c8073fe94f8ea72e27789bb9827" Nov 01 21:15:03.421785 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:03.421491 5444 scope.go:122] "RemoveContainer" containerID="8381705b6880ece47490069245d953ad11f55c9a8931f795e68cb13b82aba720" Nov 01 21:15:03.421785 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:15:03.421742 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892795-mjbq7_odh-ai-gateway-infra(95092934-7ae3-480b-8d85-f0882e449737)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" podUID="95092934-7ae3-480b-8d85-f0882e449737" Nov 01 21:15:04.427458 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:04.427429 5444 scope.go:122] "RemoveContainer" containerID="8381705b6880ece47490069245d953ad11f55c9a8931f795e68cb13b82aba720" Nov 01 21:15:04.427865 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:15:04.427705 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892795-mjbq7_odh-ai-gateway-infra(95092934-7ae3-480b-8d85-f0882e449737)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" podUID="95092934-7ae3-480b-8d85-f0882e449737" Nov 01 21:15:07.609457 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.609414 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" containerID="cri-o://076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9" gracePeriod=25 Nov 01 21:15:07.858230 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.858194 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:15:07.957881 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.957793 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume\") pod \"fa1551d5-5eee-4e58-a380-93c67298b86b\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " Nov 01 21:15:07.957881 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.957860 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume\") pod \"fa1551d5-5eee-4e58-a380-93c67298b86b\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " Nov 01 21:15:07.958099 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.957893 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials\") pod \"fa1551d5-5eee-4e58-a380-93c67298b86b\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " Nov 01 21:15:07.958099 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.957962 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv\") pod \"fa1551d5-5eee-4e58-a380-93c67298b86b\" (UID: \"fa1551d5-5eee-4e58-a380-93c67298b86b\") " Nov 01 21:15:07.958283 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.958263 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume" pod "fa1551d5-5eee-4e58-a380-93c67298b86b" (UID: "fa1551d5-5eee-4e58-a380-93c67298b86b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:07.958356 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.958293 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume" pod "fa1551d5-5eee-4e58-a380-93c67298b86b" (UID: "fa1551d5-5eee-4e58-a380-93c67298b86b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:07.960207 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.960173 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials" pod "fa1551d5-5eee-4e58-a380-93c67298b86b" (UID: "fa1551d5-5eee-4e58-a380-93c67298b86b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:07.960313 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:07.960239 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv" pod "fa1551d5-5eee-4e58-a380-93c67298b86b" (UID: "fa1551d5-5eee-4e58-a380-93c67298b86b"). InnerVolumeSpecName "kube-api-access-kzcjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:08.059541 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.059480 5444 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-plugins-config-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:08.059541 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.059530 5444 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa1551d5-5eee-4e58-a380-93c67298b86b-routing-overlay-volume\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:08.059541 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.059539 5444 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-provider-credentials\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:08.059541 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.059548 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzcjv\" (UniqueName: \"kubernetes.io/projected/fa1551d5-5eee-4e58-a380-93c67298b86b-kube-api-access-kzcjv\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:08.445549 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.445505 5444 generic.go:356] "Generic (PLEG): container finished" podID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerID="076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9" exitCode=0 Nov 01 21:15:08.445848 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.445562 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" Nov 01 21:15:08.445848 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.445596 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" event={"ID":"fa1551d5-5eee-4e58-a380-93c67298b86b","Type":"ContainerDied","Data":"076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9"} Nov 01 21:15:08.445848 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.445699 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg" event={"ID":"fa1551d5-5eee-4e58-a380-93c67298b86b","Type":"ContainerDied","Data":"aba61f33c2cabcecd55f256e343d8fa43db2db0f8bd87ce3a0bd06b22ffac9e6"} Nov 01 21:15:08.445848 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.445730 5444 scope.go:122] "RemoveContainer" containerID="076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9" Nov 01 21:15:08.454368 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.454346 5444 scope.go:122] "RemoveContainer" containerID="076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9" Nov 01 21:15:08.454726 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:15:08.454703 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9\": container with ID starting with 076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9 not found: ID does not exist" containerID="076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9" Nov 01 21:15:08.454808 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.454737 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9"} err="failed to get container status \"076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9\": rpc error: code = NotFound desc = could not find container \"076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9\": container with ID starting with 076b451c14ab11bb2b41b595507523cda3ffb1251e2b7f681c680e990dd95ce9 not found: ID does not exist" Nov 01 21:15:08.606385 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.606344 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg"] Nov 01 21:15:08.655738 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:08.655700 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6a675a/payload-processing-external-model-e2e-extmod-b-6a675a-7449zjnvg"] Nov 01 21:15:09.156676 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:09.156615 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" path="/var/lib/kubelet/pods/fa1551d5-5eee-4e58-a380-93c67298b86b/volumes" Nov 01 21:15:14.152112 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:14.152083 5444 scope.go:122] "RemoveContainer" containerID="8381705b6880ece47490069245d953ad11f55c9a8931f795e68cb13b82aba720" Nov 01 21:15:14.474879 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:14.474794 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerStarted","Data":"759a3db0460c76013f9ef69d1ef8d057aaae74988a16213b56e492ce8b4f6235"} Nov 01 21:15:15.480783 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:15.480748 5444 generic.go:356] "Generic (PLEG): container finished" podID="95092934-7ae3-480b-8d85-f0882e449737" containerID="759a3db0460c76013f9ef69d1ef8d057aaae74988a16213b56e492ce8b4f6235" exitCode=60 Nov 01 21:15:15.481262 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:15.480815 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerDied","Data":"759a3db0460c76013f9ef69d1ef8d057aaae74988a16213b56e492ce8b4f6235"} Nov 01 21:15:15.481262 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:15.480873 5444 scope.go:122] "RemoveContainer" containerID="8381705b6880ece47490069245d953ad11f55c9a8931f795e68cb13b82aba720" Nov 01 21:15:15.481262 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:15.481214 5444 scope.go:122] "RemoveContainer" containerID="759a3db0460c76013f9ef69d1ef8d057aaae74988a16213b56e492ce8b4f6235" Nov 01 21:15:15.481471 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:15:15.481447 5444 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892795-mjbq7_odh-ai-gateway-infra(95092934-7ae3-480b-8d85-f0882e449737)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" podUID="95092934-7ae3-480b-8d85-f0882e449737" Nov 01 21:15:15.596938 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:15.596907 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7"] Nov 01 21:15:16.607408 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.607383 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:16.738350 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.738255 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts\" (UniqueName: \"kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts\") pod \"95092934-7ae3-480b-8d85-f0882e449737\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " Nov 01 21:15:16.738350 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.738301 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca\") pod \"95092934-7ae3-480b-8d85-f0882e449737\" (UID: \"95092934-7ae3-480b-8d85-f0882e449737\") " Nov 01 21:15:16.738791 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.738755 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca" pod "95092934-7ae3-480b-8d85-f0882e449737" (UID: "95092934-7ae3-480b-8d85-f0882e449737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:15:16.740610 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.740572 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts" pod "95092934-7ae3-480b-8d85-f0882e449737" (UID: "95092934-7ae3-480b-8d85-f0882e449737"). InnerVolumeSpecName "kube-api-access-g2jts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:15:16.839409 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.839364 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95092934-7ae3-480b-8d85-f0882e449737-maas-api-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:16.839409 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:16.839398 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jts\" (UniqueName: \"kubernetes.io/projected/95092934-7ae3-480b-8d85-f0882e449737-kube-api-access-g2jts\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:15:17.492276 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:17.492186 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" event={"ID":"95092934-7ae3-480b-8d85-f0882e449737","Type":"ContainerDied","Data":"0cb3c77e4ef35547d2867a2993c7217e80cac8904b7b3e769055c08d7bc8ad7b"} Nov 01 21:15:17.492276 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:17.492213 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7" Nov 01 21:15:17.492276 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:17.492251 5444 scope.go:122] "RemoveContainer" containerID="759a3db0460c76013f9ef69d1ef8d057aaae74988a16213b56e492ce8b4f6235" Nov 01 21:15:17.655440 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:17.655405 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7"] Nov 01 21:15:17.658580 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:17.658553 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892795-mjbq7"] Nov 01 21:15:19.157568 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:19.157528 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95092934-7ae3-480b-8d85-f0882e449737" path="/var/lib/kubelet/pods/95092934-7ae3-480b-8d85-f0882e449737/volumes" Nov 01 21:15:30.035071 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:15:30.035036 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:15:48.541961 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:48.541927 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vnw68_1c283462-a39d-406e-be48-ad3c33a27857/kuadrant-console-plugin/0.log" Nov 01 21:15:52.448716 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:15:52.448677 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qh45n_88630e47-4588-44a1-94ac-f39973ad1406/manager/0.log" Nov 01 21:16:00.035823 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:16:00.035783 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:16:30.036562 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:16:30.036521 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:17:00.034939 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:17:00.034845 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:17:30.034569 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:17:30.034531 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:18:00.036770 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:18:00.036724 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:18:30.033917 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:18:30.033877 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.564259 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl"] Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.564923 5444 cpu_manager.go:414] "containerMap: removing container" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.564942 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.564971 5444 cpu_manager.go:414] "containerMap: removing container" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.564980 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.565213 5444 cpu_manager.go:414] "containerMap: removing container" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.565228 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.565410 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.565429 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1551d5-5eee-4e58-a380-93c67298b86b" containerName="payload-processing" Nov 01 21:18:36.566894 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.565443 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.569471 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.569034 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.612475 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.612448 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5"] Nov 01 21:18:36.612822 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.612808 5444 cpu_manager.go:414] "containerMap: removing container" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.612865 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.612823 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.612903 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.612889 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="95092934-7ae3-480b-8d85-f0882e449737" containerName="cleanup" Nov 01 21:18:36.615811 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.615792 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.661312 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661290 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661414 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661322 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661414 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661343 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79bgm\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661490 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661408 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661490 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661455 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661557 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661495 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661557 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661513 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661557 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661530 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.661675 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.661630 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.669965 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.669941 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl"] Nov 01 21:18:36.728015 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.727979 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5"] Nov 01 21:18:36.762107 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762083 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762200 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762118 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8fl\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762200 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762136 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762200 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762159 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762200 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762193 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762346 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762210 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762346 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762234 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762346 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762305 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762451 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762363 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762451 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762430 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762524 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762472 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762562 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762521 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762562 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762553 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762587 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762616 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762656 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79bgm\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762709 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762686 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.762968 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762758 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.762968 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762791 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.763079 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.762964 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.763230 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.763210 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.763299 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.763228 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.763299 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.763209 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.764668 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.764624 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.764891 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.764872 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.818854 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.818794 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.818854 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.818806 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79bgm\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-jdqnl\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.863702 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863676 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.863810 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863706 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.863810 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863750 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.863810 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863772 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.863810 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863794 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864062 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863811 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8fl\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864062 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863831 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864062 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863868 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864062 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.863894 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864331 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.864169 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864396 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.864338 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864396 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.864349 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864514 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.864448 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.864514 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.864500 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.865823 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.865806 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.866193 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.866176 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.881075 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.881059 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:36.919530 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.919499 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.919705 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.919559 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8fl\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8n8s5\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:36.927365 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:36.927336 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:37.303871 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.303831 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl"] Nov 01 21:18:37.347498 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:18:37.347474 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda924e870_cc94_41ba_8782_0a820814a41e.slice/crio-b14eda4a7628b10ef792a79978c5966e675de18c4b14dd7adceaea6831aeef27 WatchSource:0}: Error finding container b14eda4a7628b10ef792a79978c5966e675de18c4b14dd7adceaea6831aeef27: Status 404 returned error can't find the container with id b14eda4a7628b10ef792a79978c5966e675de18c4b14dd7adceaea6831aeef27 Nov 01 21:18:37.349508 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.349475 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:37.349612 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.349540 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:37.349612 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.349568 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:37.358018 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.357979 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5"] Nov 01 21:18:37.360993 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:18:37.360970 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab2ac29_924b_4cf1_bea6_cc72a48dd052.slice/crio-b8d0eb6e28c26085148cd2da14e2ed27db4d43a1fb5f0fe5befffdbd1f29e13d WatchSource:0}: Error finding container b8d0eb6e28c26085148cd2da14e2ed27db4d43a1fb5f0fe5befffdbd1f29e13d: Status 404 returned error can't find the container with id b8d0eb6e28c26085148cd2da14e2ed27db4d43a1fb5f0fe5befffdbd1f29e13d Nov 01 21:18:37.363060 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.363034 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:37.363177 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.363086 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:37.363177 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:37.363112 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:18:38.303477 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.302620 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" event={"ID":"1ab2ac29-924b-4cf1-bea6-cc72a48dd052","Type":"ContainerStarted","Data":"11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59"} Nov 01 21:18:38.303477 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.302708 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" event={"ID":"1ab2ac29-924b-4cf1-bea6-cc72a48dd052","Type":"ContainerStarted","Data":"b8d0eb6e28c26085148cd2da14e2ed27db4d43a1fb5f0fe5befffdbd1f29e13d"} Nov 01 21:18:38.305046 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.305017 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" event={"ID":"a924e870-cc94-41ba-8782-0a820814a41e","Type":"ContainerStarted","Data":"f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6"} Nov 01 21:18:38.305184 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.305057 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" event={"ID":"a924e870-cc94-41ba-8782-0a820814a41e","Type":"ContainerStarted","Data":"b14eda4a7628b10ef792a79978c5966e675de18c4b14dd7adceaea6831aeef27"} Nov 01 21:18:38.426328 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.426266 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podStartSLOduration=2.426254238 podStartE2EDuration="2.426254238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:38.425284772 +0000 UTC m=+4519.370122643" watchObservedRunningTime="2026-11-01 21:18:38.426254238 +0000 UTC m=+4519.371092102" Nov 01 21:18:38.543187 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.543126 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podStartSLOduration=2.543111768 podStartE2EDuration="2.543111768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:18:38.542405922 +0000 UTC m=+4519.487243815" watchObservedRunningTime="2026-11-01 21:18:38.543111768 +0000 UTC m=+4519.487949635" Nov 01 21:18:38.884985 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.884436 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:38.884985 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.884623 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:38.884985 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.884691 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:38.928772 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.928734 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:38.929092 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.929070 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:38.929142 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:38.929117 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:39.885011 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:39.884977 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:39.885622 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:39.885583 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:39.928236 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:39.928134 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:39.928236 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:39.928182 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:40.882041 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:40.882006 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:40.882433 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:40.882403 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:40.928098 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:40.928069 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:40.928456 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:40.928114 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:41.885050 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:41.885010 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:41.885344 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:41.885311 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:41.928701 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:41.928672 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:41.929052 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:41.928714 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:42.885661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:42.885030 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:42.885661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:42.885095 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:42.928921 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:42.928885 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:42.929292 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:42.928935 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:43.881757 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:43.881726 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:43.881933 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:43.881779 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:43.928422 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:43.928389 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:43.928533 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:43.928443 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:44.881402 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:44.881371 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 21:18:44.881819 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:44.881421 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 21:18:44.928157 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:44.928127 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8n8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 01 21:18:44.928290 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:44.928182 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 01 21:18:46.593238 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:46.593190 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:46.594138 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:46.594113 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:46.594852 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:46.594830 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:18:46.608979 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:46.608960 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:46.609240 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:46.609219 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:46.609899 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:46.609878 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:18:51.574843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:51.574799 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:18:51.575265 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:18:51.574902 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:19:00.034548 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:19:00.034513 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:19:30.033201 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:19:30.033165 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:20:00.033796 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:20:00.033723 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:20:20.197655 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:20:20.197607 5444 scope.go:122] "RemoveContainer" containerID="397a88a64ee68012a8c01c0a8f82404fae7c7cd8087aca540809f2c3f7f61ba7" Nov 01 21:20:30.034259 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:20:30.034217 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:21:00.034741 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:21:00.034707 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:21:30.034822 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:21:30.034785 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:22:00.034112 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:22:00.034076 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:22:30.035743 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:22:30.035697 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:23:00.034540 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:23:00.034461 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:23:30.035475 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:23:30.035432 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:24:00.036799 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:24:00.036753 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:24:30.033486 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:24:30.033449 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:24:45.251698 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.251668 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2"] Nov 01 21:24:45.255047 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.255028 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.356454 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.356428 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wskl6\"" Nov 01 21:24:45.356454 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.356428 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 21:24:45.356708 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.356694 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 21:24:45.356779 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.356708 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9cd91242-kserve-self-signed-certs\"" Nov 01 21:24:45.357717 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.357697 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2"] Nov 01 21:24:45.388457 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.388433 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.388559 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.388467 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.388559 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.388486 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfb5\" (UniqueName: \"kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.388679 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.388656 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.388722 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.388688 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.388722 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.388714 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.489655 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.489600 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.489655 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.489653 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.489873 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.489672 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.489873 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.489695 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.489873 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.489724 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.489873 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.489742 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfb5\" (UniqueName: \"kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.490135 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.490108 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.490197 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.490151 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.490197 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.490166 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.491928 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.491907 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.492138 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.492118 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.545492 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.545465 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfb5\" (UniqueName: \"kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5\") pod \"e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.566172 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.566154 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:45.941102 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.941032 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2"] Nov 01 21:24:45.944495 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:24:45.944467 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57c7b2b7_e280_43a9_b501_799aad9a8702.slice/crio-adc2b3b664c3916e838b602a573b5917ffd4ca2299427ba7dac693ca06ee6053 WatchSource:0}: Error finding container adc2b3b664c3916e838b602a573b5917ffd4ca2299427ba7dac693ca06ee6053: Status 404 returned error can't find the container with id adc2b3b664c3916e838b602a573b5917ffd4ca2299427ba7dac693ca06ee6053 Nov 01 21:24:45.994083 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.994059 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" event={"ID":"57c7b2b7-e280-43a9-b501-799aad9a8702","Type":"ContainerStarted","Data":"89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e"} Nov 01 21:24:45.994183 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:45.994095 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" event={"ID":"57c7b2b7-e280-43a9-b501-799aad9a8702","Type":"ContainerStarted","Data":"adc2b3b664c3916e838b602a573b5917ffd4ca2299427ba7dac693ca06ee6053"} Nov 01 21:24:47.111756 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:47.111699 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" podStartSLOduration=2.111680865 podStartE2EDuration="2.111680865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:24:47.111361046 +0000 UTC m=+4888.056198915" watchObservedRunningTime="2026-11-01 21:24:47.111680865 +0000 UTC m=+4888.056518734" Nov 01 21:24:47.567108 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:47.567075 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:55.566386 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:55.566352 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:55.578866 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:55.578841 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:24:56.050465 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:24:56.050432 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:25:00.034269 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:25:00.034233 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:25:29.764123 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:29.764088 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2"] Nov 01 21:25:30.034161 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:25:30.034079 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:25:36.575799 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:36.575754 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:25:36.576461 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:36.576427 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:25:44.783276 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:44.783237 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" podUID="57c7b2b7-e280-43a9-b501-799aad9a8702" containerName="main" containerID="cri-o://89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e" gracePeriod=45 Nov 01 21:25:45.023254 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.023228 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:25:45.125876 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.125788 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5\" (UniqueName: \"kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5\") pod \"57c7b2b7-e280-43a9-b501-799aad9a8702\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " Nov 01 21:25:45.125876 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.125828 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm\") pod \"57c7b2b7-e280-43a9-b501-799aad9a8702\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " Nov 01 21:25:45.125876 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.125871 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache\") pod \"57c7b2b7-e280-43a9-b501-799aad9a8702\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " Nov 01 21:25:45.126181 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.125895 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home\") pod \"57c7b2b7-e280-43a9-b501-799aad9a8702\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " Nov 01 21:25:45.126181 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.125919 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs\" (UniqueName: \"kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs\") pod \"57c7b2b7-e280-43a9-b501-799aad9a8702\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " Nov 01 21:25:45.126181 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.125966 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir\") pod \"57c7b2b7-e280-43a9-b501-799aad9a8702\" (UID: \"57c7b2b7-e280-43a9-b501-799aad9a8702\") " Nov 01 21:25:45.126311 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.126172 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home" pod "57c7b2b7-e280-43a9-b501-799aad9a8702" (UID: "57c7b2b7-e280-43a9-b501-799aad9a8702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:45.126311 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.126188 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache" pod "57c7b2b7-e280-43a9-b501-799aad9a8702" (UID: "57c7b2b7-e280-43a9-b501-799aad9a8702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:45.126406 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.126298 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir" pod "57c7b2b7-e280-43a9-b501-799aad9a8702" (UID: "57c7b2b7-e280-43a9-b501-799aad9a8702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:45.128092 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.128067 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm" pod "57c7b2b7-e280-43a9-b501-799aad9a8702" (UID: "57c7b2b7-e280-43a9-b501-799aad9a8702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:25:45.128190 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.128097 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs" pod "57c7b2b7-e280-43a9-b501-799aad9a8702" (UID: "57c7b2b7-e280-43a9-b501-799aad9a8702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:25:45.128248 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.128206 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5" pod "57c7b2b7-e280-43a9-b501-799aad9a8702" (UID: "57c7b2b7-e280-43a9-b501-799aad9a8702"). InnerVolumeSpecName "kube-api-access-sjfb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:25:45.227393 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.227359 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:45.227393 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.227386 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjfb5\" (UniqueName: \"kubernetes.io/projected/57c7b2b7-e280-43a9-b501-799aad9a8702-kube-api-access-sjfb5\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:45.227393 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.227396 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:45.227612 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.227405 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:45.227612 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.227413 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c7b2b7-e280-43a9-b501-799aad9a8702-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:45.227612 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.227421 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c7b2b7-e280-43a9-b501-799aad9a8702-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:25:45.267530 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.267494 5444 generic.go:356] "Generic (PLEG): container finished" podID="57c7b2b7-e280-43a9-b501-799aad9a8702" containerID="89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e" exitCode=0 Nov 01 21:25:45.267730 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.267540 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" Nov 01 21:25:45.267730 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.267573 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" event={"ID":"57c7b2b7-e280-43a9-b501-799aad9a8702","Type":"ContainerDied","Data":"89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e"} Nov 01 21:25:45.267730 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.267616 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2" event={"ID":"57c7b2b7-e280-43a9-b501-799aad9a8702","Type":"ContainerDied","Data":"adc2b3b664c3916e838b602a573b5917ffd4ca2299427ba7dac693ca06ee6053"} Nov 01 21:25:45.267730 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.267632 5444 scope.go:122] "RemoveContainer" containerID="89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e" Nov 01 21:25:45.275610 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.275591 5444 scope.go:122] "RemoveContainer" containerID="89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e" Nov 01 21:25:45.276163 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:25:45.276136 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e\": container with ID starting with 89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e not found: ID does not exist" containerID="89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e" Nov 01 21:25:45.276261 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.276162 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e"} err="failed to get container status \"89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e\": rpc error: code = NotFound desc = could not find container \"89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e\": container with ID starting with 89b6d537fe791b10913d2e3821061838522610b23baf43ec1078e12b1404597e not found: ID does not exist" Nov 01 21:25:45.428314 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.428215 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2"] Nov 01 21:25:45.431454 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:45.431430 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9cd91242-kserve-c9d94b7d9-nxvw2"] Nov 01 21:25:47.162022 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:25:47.161986 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c7b2b7-e280-43a9-b501-799aad9a8702" path="/var/lib/kubelet/pods/57c7b2b7-e280-43a9-b501-799aad9a8702/volumes" Nov 01 21:26:00.034360 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:26:00.034326 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:26:30.034203 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:26:30.034166 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:27:00.034212 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:27:00.034174 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:27:30.033962 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:27:30.033924 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:27:52.108857 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.108814 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf"] Nov 01 21:27:52.109237 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.109131 5444 cpu_manager.go:414] "containerMap: removing container" podUID="57c7b2b7-e280-43a9-b501-799aad9a8702" containerName="main" Nov 01 21:27:52.109237 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.109141 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c7b2b7-e280-43a9-b501-799aad9a8702" containerName="main" Nov 01 21:27:52.109237 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.109204 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c7b2b7-e280-43a9-b501-799aad9a8702" containerName="main" Nov 01 21:27:52.112107 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.112088 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.163738 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.163714 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-958b82\"/\"kube-root-ca.crt\"" Nov 01 21:27:52.213132 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.213105 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-958b82\"/\"openshift-service-ca.crt\"" Nov 01 21:27:52.213279 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.213250 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-958b82\"/\"default-dockercfg-7pbz6\"" Nov 01 21:27:52.213352 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.213268 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-958b82\"/\"ipp-route-958b82-kserve-self-signed-certs\"" Nov 01 21:27:52.214231 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.214207 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf"] Nov 01 21:27:52.251734 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.251712 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.251840 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.251822 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.251908 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.251863 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.251980 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.251959 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.252045 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.252005 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz865\" (UniqueName: \"kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.252111 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.252086 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.352706 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.352666 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz865\" (UniqueName: \"kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.352909 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.352724 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.352909 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.352767 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.352909 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.352831 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.352909 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.352871 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.353143 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.352916 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.353242 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.353220 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.353297 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.353259 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.353297 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.353230 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.355167 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.355142 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.355555 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.355534 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.408375 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.408314 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz865\" (UniqueName: \"kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865\") pod \"ipp-route-958b82-kserve-7b757d97cc-mmxpf\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.423183 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.423154 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:27:52.793771 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.793743 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf"] Nov 01 21:27:52.797140 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:27:52.797112 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc854c0d2_df49_4e81_b1d4_d2baad8098b4.slice/crio-678d1bb018699269afd51a5722ad72ca49aa3633d96c072e63aa909813bc1790 WatchSource:0}: Error finding container 678d1bb018699269afd51a5722ad72ca49aa3633d96c072e63aa909813bc1790: Status 404 returned error can't find the container with id 678d1bb018699269afd51a5722ad72ca49aa3633d96c072e63aa909813bc1790 Nov 01 21:27:52.852450 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:52.852427 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" event={"ID":"c854c0d2-df49-4e81-b1d4-d2baad8098b4","Type":"ContainerStarted","Data":"678d1bb018699269afd51a5722ad72ca49aa3633d96c072e63aa909813bc1790"} Nov 01 21:27:53.857959 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:53.857925 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" event={"ID":"c854c0d2-df49-4e81-b1d4-d2baad8098b4","Type":"ContainerStarted","Data":"b8cdce8b1eca3f3967bdcc252a59afa0e8f167d87bd6e5c65134e77338f17ed8"} Nov 01 21:27:53.969788 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:53.969739 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" podStartSLOduration=1.969723163 podStartE2EDuration="1.969723163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:27:53.968957483 +0000 UTC m=+5074.913795365" watchObservedRunningTime="2026-11-01 21:27:53.969723163 +0000 UTC m=+5074.914561022" Nov 01 21:27:54.424111 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:27:54.424082 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:28:00.033111 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:28:00.033078 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:28:02.424256 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:28:02.424224 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:28:02.436698 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:28:02.436662 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:28:02.907889 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:28:02.907864 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:28:30.035053 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:28:30.035016 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:28:54.960575 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:28:54.960489 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf"] Nov 01 21:29:00.034155 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:29:00.034120 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:29:09.979676 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:09.979604 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" podUID="c854c0d2-df49-4e81-b1d4-d2baad8098b4" containerName="main" containerID="cri-o://b8cdce8b1eca3f3967bdcc252a59afa0e8f167d87bd6e5c65134e77338f17ed8" gracePeriod=45 Nov 01 21:29:10.181582 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.181550 5444 generic.go:356] "Generic (PLEG): container finished" podID="c854c0d2-df49-4e81-b1d4-d2baad8098b4" containerID="b8cdce8b1eca3f3967bdcc252a59afa0e8f167d87bd6e5c65134e77338f17ed8" exitCode=0 Nov 01 21:29:10.181735 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.181618 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" event={"ID":"c854c0d2-df49-4e81-b1d4-d2baad8098b4","Type":"ContainerDied","Data":"b8cdce8b1eca3f3967bdcc252a59afa0e8f167d87bd6e5c65134e77338f17ed8"} Nov 01 21:29:10.218147 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.218127 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:29:10.236358 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236319 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home\") pod \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " Nov 01 21:29:10.236438 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236375 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir\") pod \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " Nov 01 21:29:10.236527 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236509 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache\") pod \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " Nov 01 21:29:10.236589 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236545 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm\") pod \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " Nov 01 21:29:10.236660 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236593 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865\" (UniqueName: \"kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865\") pod \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " Nov 01 21:29:10.236660 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236593 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir" pod "c854c0d2-df49-4e81-b1d4-d2baad8098b4" (UID: "c854c0d2-df49-4e81-b1d4-d2baad8098b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:29:10.236779 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236661 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs\") pod \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\" (UID: \"c854c0d2-df49-4e81-b1d4-d2baad8098b4\") " Nov 01 21:29:10.236846 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236606 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home" pod "c854c0d2-df49-4e81-b1d4-d2baad8098b4" (UID: "c854c0d2-df49-4e81-b1d4-d2baad8098b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:29:10.236846 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236749 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache" pod "c854c0d2-df49-4e81-b1d4-d2baad8098b4" (UID: "c854c0d2-df49-4e81-b1d4-d2baad8098b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:29:10.236954 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236929 5444 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tmp-dir\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:10.237020 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236956 5444 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-model-cache\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:10.237020 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.236971 5444 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-home\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:10.238843 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.238805 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs" pod "c854c0d2-df49-4e81-b1d4-d2baad8098b4" (UID: "c854c0d2-df49-4e81-b1d4-d2baad8098b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:29:10.238942 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.238872 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865" pod "c854c0d2-df49-4e81-b1d4-d2baad8098b4" (UID: "c854c0d2-df49-4e81-b1d4-d2baad8098b4"). InnerVolumeSpecName "kube-api-access-tz865". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:29:10.238942 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.238881 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm" pod "c854c0d2-df49-4e81-b1d4-d2baad8098b4" (UID: "c854c0d2-df49-4e81-b1d4-d2baad8098b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:29:10.337713 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.337693 5444 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c854c0d2-df49-4e81-b1d4-d2baad8098b4-dshm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:10.337816 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.337715 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz865\" (UniqueName: \"kubernetes.io/projected/c854c0d2-df49-4e81-b1d4-d2baad8098b4-kube-api-access-tz865\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:10.337816 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:10.337730 5444 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c854c0d2-df49-4e81-b1d4-d2baad8098b4-tls-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:29:11.185968 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:11.185931 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" event={"ID":"c854c0d2-df49-4e81-b1d4-d2baad8098b4","Type":"ContainerDied","Data":"678d1bb018699269afd51a5722ad72ca49aa3633d96c072e63aa909813bc1790"} Nov 01 21:29:11.185968 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:11.185962 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf" Nov 01 21:29:11.185968 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:11.185973 5444 scope.go:122] "RemoveContainer" containerID="b8cdce8b1eca3f3967bdcc252a59afa0e8f167d87bd6e5c65134e77338f17ed8" Nov 01 21:29:11.345447 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:11.345423 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf"] Nov 01 21:29:11.394616 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:11.394593 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-958b82/ipp-route-958b82-kserve-7b757d97cc-mmxpf"] Nov 01 21:29:13.156878 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:29:13.156846 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c854c0d2-df49-4e81-b1d4-d2baad8098b4" path="/var/lib/kubelet/pods/c854c0d2-df49-4e81-b1d4-d2baad8098b4/volumes" Nov 01 21:29:30.033291 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:29:30.033255 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:30:00.034739 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:30:00.034698 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:30:30.035352 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:30:30.035318 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:31:00.034265 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:31:00.034231 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:31:30.036232 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:31:30.036194 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:32:00.035614 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:32:00.035582 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:32:30.033292 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:32:30.033255 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:33:00.033789 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:33:00.033746 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:33:21.595662 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:33:21.594742 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:33:21.595662 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:33:21.594828 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:33:30.035044 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:33:30.035005 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:34:00.037558 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:34:00.037518 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:34:30.035652 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:34:30.035599 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:35:00.034026 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:35:00.033949 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:35:30.034333 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:35:30.034293 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:36:00.034053 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:36:00.034018 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:36:30.034338 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:36:30.034303 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:37:00.041275 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:37:00.041222 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:37:02.658788 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:37:02.658727 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-jdqnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:37:02.659563 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:37:02.659525 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:37:30.039562 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:37:30.039520 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:38:00.037566 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:38:00.037475 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:38:06.580851 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:38:06.580772 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:38:06.581901 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:38:06.581862 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:38:30.033352 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:38:30.033316 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:38:36.583661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:38:36.582757 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 21:38:36.583661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:38:36.582844 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 21:39:00.033161 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:39:00.033125 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:39:30.033505 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:39:30.033462 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:40:00.035912 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:40:00.035854 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:40:30.036576 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:40:30.036531 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:41:00.034014 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:41:00.033941 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:41:30.034026 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:41:30.033979 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:42:00.038084 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:42:00.038042 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:42:30.036818 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:42:30.036773 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:43:00.034737 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:43:00.034696 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:43:30.043123 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:43:30.043088 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:43:52.907859 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:52.907781 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl"] Nov 01 21:43:52.908334 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:52.908118 5444 cpu_manager.go:414] "containerMap: removing container" podUID="c854c0d2-df49-4e81-b1d4-d2baad8098b4" containerName="main" Nov 01 21:43:52.908334 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:52.908129 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c854c0d2-df49-4e81-b1d4-d2baad8098b4" containerName="main" Nov 01 21:43:52.908334 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:52.908197 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="c854c0d2-df49-4e81-b1d4-d2baad8098b4" containerName="main" Nov 01 21:43:52.911327 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:52.911305 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.014117 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.014084 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl"] Nov 01 21:43:53.021504 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.021477 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f82cfd-0abd-4086-8577-d059fa8d9d82-maas-api-tls\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.021628 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.021531 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wd7c\" (UniqueName: \"kubernetes.io/projected/73f82cfd-0abd-4086-8577-d059fa8d9d82-kube-api-access-7wd7c\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.021688 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.021664 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f82cfd-0abd-4086-8577-d059fa8d9d82-openshift-service-ca\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.021742 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.021725 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f82cfd-0abd-4086-8577-d059fa8d9d82-metrics-certs\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.122825 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.122791 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wd7c\" (UniqueName: \"kubernetes.io/projected/73f82cfd-0abd-4086-8577-d059fa8d9d82-kube-api-access-7wd7c\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.122825 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.122830 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f82cfd-0abd-4086-8577-d059fa8d9d82-openshift-service-ca\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.123022 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.122897 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f82cfd-0abd-4086-8577-d059fa8d9d82-metrics-certs\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.123022 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.122977 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f82cfd-0abd-4086-8577-d059fa8d9d82-maas-api-tls\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.123739 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.123716 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f82cfd-0abd-4086-8577-d059fa8d9d82-openshift-service-ca\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.125319 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.125294 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f82cfd-0abd-4086-8577-d059fa8d9d82-maas-api-tls\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.125450 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.125385 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f82cfd-0abd-4086-8577-d059fa8d9d82-metrics-certs\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.179412 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.179338 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wd7c\" (UniqueName: \"kubernetes.io/projected/73f82cfd-0abd-4086-8577-d059fa8d9d82-kube-api-access-7wd7c\") pod \"maas-api-8595f64f4d-grjkl\" (UID: \"73f82cfd-0abd-4086-8577-d059fa8d9d82\") " pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.223307 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.223266 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:53.593586 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.593547 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl"] Nov 01 21:43:53.597385 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:43:53.597357 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f82cfd_0abd_4086_8577_d059fa8d9d82.slice/crio-44e99e2b1adbfccbe36c94e6b620c3c5360669f3fee04894a6a9bee67850d296 WatchSource:0}: Error finding container 44e99e2b1adbfccbe36c94e6b620c3c5360669f3fee04894a6a9bee67850d296: Status 404 returned error can't find the container with id 44e99e2b1adbfccbe36c94e6b620c3c5360669f3fee04894a6a9bee67850d296 Nov 01 21:43:53.598734 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:53.598717 5444 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 21:43:54.015177 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:54.015144 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" event={"ID":"73f82cfd-0abd-4086-8577-d059fa8d9d82","Type":"ContainerStarted","Data":"44e99e2b1adbfccbe36c94e6b620c3c5360669f3fee04894a6a9bee67850d296"} Nov 01 21:43:55.020655 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:55.020607 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" event={"ID":"73f82cfd-0abd-4086-8577-d059fa8d9d82","Type":"ContainerStarted","Data":"4d0239e5d3183a26fd798d91bb846e1fc8e5c7763494cf88bd475116ff408119"} Nov 01 21:43:55.021042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:55.020759 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:43:55.134402 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:43:55.134345 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" podStartSLOduration=2.231518825 podStartE2EDuration="3.134328607s" totalImagesPullingTime="902.809782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:55.133254695 +0000 UTC m=+6036.078092572" watchObservedRunningTime="2026-11-01 21:43:55.134328607 +0000 UTC m=+6036.079166466" Nov 01 21:44:00.040999 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:44:00.040414 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:44:01.030720 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:01.030691 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8595f64f4d-grjkl" Nov 01 21:44:01.194379 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:01.194346 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7"] Nov 01 21:44:01.194840 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:01.194547 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" podUID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" containerName="maas-api" containerID="cri-o://ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92" gracePeriod=30 Nov 01 21:44:02.438308 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.438284 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 21:44:02.534336 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.534243 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls\") pod \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " Nov 01 21:44:02.534336 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.534297 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj\" (UniqueName: \"kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj\") pod \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " Nov 01 21:44:02.534336 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.534330 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs\") pod \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " Nov 01 21:44:02.534629 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.534427 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca\") pod \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\" (UID: \"c9b1cc77-92f0-4160-8f97-27d1e736ef46\") " Nov 01 21:44:02.534839 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.534812 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca" pod "c9b1cc77-92f0-4160-8f97-27d1e736ef46" (UID: "c9b1cc77-92f0-4160-8f97-27d1e736ef46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:44:02.536478 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.536447 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls" pod "c9b1cc77-92f0-4160-8f97-27d1e736ef46" (UID: "c9b1cc77-92f0-4160-8f97-27d1e736ef46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:44:02.536575 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.536537 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs" pod "c9b1cc77-92f0-4160-8f97-27d1e736ef46" (UID: "c9b1cc77-92f0-4160-8f97-27d1e736ef46"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 21:44:02.536575 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.536543 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj" pod "c9b1cc77-92f0-4160-8f97-27d1e736ef46" (UID: "c9b1cc77-92f0-4160-8f97-27d1e736ef46"). InnerVolumeSpecName "kube-api-access-hdghj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:44:02.635949 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.635915 5444 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-maas-api-tls\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:44:02.636149 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.635966 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdghj\" (UniqueName: \"kubernetes.io/projected/c9b1cc77-92f0-4160-8f97-27d1e736ef46-kube-api-access-hdghj\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:44:02.636149 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.635982 5444 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9b1cc77-92f0-4160-8f97-27d1e736ef46-metrics-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:44:02.636149 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:02.635993 5444 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b1cc77-92f0-4160-8f97-27d1e736ef46-openshift-service-ca\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:44:03.055908 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.055875 5444 generic.go:356] "Generic (PLEG): container finished" podID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" containerID="ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92" exitCode=0 Nov 01 21:44:03.056085 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.055945 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" Nov 01 21:44:03.056085 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.055956 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" event={"ID":"c9b1cc77-92f0-4160-8f97-27d1e736ef46","Type":"ContainerDied","Data":"ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92"} Nov 01 21:44:03.056085 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.056034 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" event={"ID":"c9b1cc77-92f0-4160-8f97-27d1e736ef46","Type":"ContainerDied","Data":"ac9fe528c45694d8f51183be173e0cc7c28bd41fd806bf7c940c8fc02ef133bb"} Nov 01 21:44:03.056085 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.056058 5444 scope.go:122] "RemoveContainer" containerID="ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92" Nov 01 21:44:03.065517 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.065502 5444 scope.go:122] "RemoveContainer" containerID="ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92" Nov 01 21:44:03.065812 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:44:03.065793 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92\": container with ID starting with ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92 not found: ID does not exist" containerID="ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92" Nov 01 21:44:03.065892 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.065824 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92"} err="failed to get container status \"ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92\": rpc error: code = NotFound desc = could not find container \"ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92\": container with ID starting with ea17895349176282dcca12cf8ec34d92660a1e342038978e65b48db483284a92 not found: ID does not exist" Nov 01 21:44:03.218895 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.218875 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7"] Nov 01 21:44:03.268150 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:03.268127 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7"] Nov 01 21:44:05.157045 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:05.157000 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" path="/var/lib/kubelet/pods/c9b1cc77-92f0-4160-8f97-27d1e736ef46/volumes" Nov 01 21:44:05.388771 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:44:05.388724 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-845db7b4d-d4dc7" podUID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.25:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 21:44:30.033938 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:44:30.033905 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:45:00.033448 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:45:00.033409 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:45:30.034870 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:45:30.034831 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:46:00.037265 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:46:00.037222 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:46:30.035110 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:46:30.035073 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:47:00.034518 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:47:00.034441 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:47:30.034981 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:47:30.034943 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:48:00.036186 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:48:00.036140 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:48:30.035706 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:48:30.035669 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:49:00.033840 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:49:00.033801 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:49:30.033501 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:49:30.033467 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:50:00.037469 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:50:00.037384 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:50:30.033575 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:50:30.033540 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:51:00.033654 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:51:00.033606 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:51:30.034495 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:51:30.034458 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:52:00.034148 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:52:00.034111 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:52:30.038163 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:52:30.038124 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:52:51.383621 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:51.383546 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl"] Nov 01 21:52:51.384121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:51.383773 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" containerID="cri-o://f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6" gracePeriod=30 Nov 01 21:52:51.432980 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:51.432952 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5"] Nov 01 21:52:51.433154 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:51.433132 5444 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" containerID="cri-o://11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59" gracePeriod=30 Nov 01 21:52:56.673272 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.673220 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:52:56.696058 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.696035 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:52:56.762820 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762786 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762838 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762859 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762887 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762904 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762923 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762949 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763004 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.762977 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763008 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763041 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763113 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763119 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763147 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763174 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data\") pod \"a924e870-cc94-41ba-8782-0a820814a41e\" (UID: \"a924e870-cc94-41ba-8782-0a820814a41e\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763202 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763215 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763230 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763254 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763298 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763331 5444 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert\") pod \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\" (UID: \"1ab2ac29-924b-4cf1-bea6-cc72a48dd052\") " Nov 01 21:52:56.763397 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763333 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763549 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763601 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763625 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763666 5444 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-certs\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.763854 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.764048 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.764121 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.764056 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 21:52:56.764952 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.764926 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.765070 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.764992 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.765070 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.765047 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.765529 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.765501 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.765827 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.765800 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "kube-api-access-79bgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.766234 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.766203 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:52:56.766318 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.766231 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 21:52:56.766318 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.766296 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.766536 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.766514 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token" pod "a924e870-cc94-41ba-8782-0a820814a41e" (UID: "a924e870-cc94-41ba-8782-0a820814a41e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.766766 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.766738 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 21:52:56.766867 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.766851 5444 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl" pod "1ab2ac29-924b-4cf1-bea6-cc72a48dd052" (UID: "1ab2ac29-924b-4cf1-bea6-cc72a48dd052"). InnerVolumeSpecName "kube-api-access-xg8fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 21:52:56.864791 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864761 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79bgm\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-kube-api-access-79bgm\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.864791 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864786 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.864791 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864795 5444 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-credential-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864803 5444 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-workload-socket\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864811 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a924e870-cc94-41ba-8782-0a820814a41e-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864825 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864837 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864845 5444 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg8fl\" (UniqueName: \"kubernetes.io/projected/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-kube-api-access-xg8fl\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864854 5444 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-podinfo\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864863 5444 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-data\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864871 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864879 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a924e870-cc94-41ba-8782-0a820814a41e-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864886 5444 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a924e870-cc94-41ba-8782-0a820814a41e-istio-token\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864894 5444 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a924e870-cc94-41ba-8782-0a820814a41e-istiod-ca-cert\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:56.865014 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:56.864902 5444 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ab2ac29-924b-4cf1-bea6-cc72a48dd052-istio-envoy\") on node \"ip-10-0-138-177.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 21:52:57.199485 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.199445 5444 generic.go:356] "Generic (PLEG): container finished" podID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerID="11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59" exitCode=0 Nov 01 21:52:57.199660 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.199556 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" Nov 01 21:52:57.199660 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.199563 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" event={"ID":"1ab2ac29-924b-4cf1-bea6-cc72a48dd052","Type":"ContainerDied","Data":"11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59"} Nov 01 21:52:57.199818 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.199690 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5" event={"ID":"1ab2ac29-924b-4cf1-bea6-cc72a48dd052","Type":"ContainerDied","Data":"b8d0eb6e28c26085148cd2da14e2ed27db4d43a1fb5f0fe5befffdbd1f29e13d"} Nov 01 21:52:57.199818 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.199746 5444 scope.go:122] "RemoveContainer" containerID="11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59" Nov 01 21:52:57.201016 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.200992 5444 generic.go:356] "Generic (PLEG): container finished" podID="a924e870-cc94-41ba-8782-0a820814a41e" containerID="f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6" exitCode=0 Nov 01 21:52:57.201090 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.201047 5444 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" Nov 01 21:52:57.201149 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.201044 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" event={"ID":"a924e870-cc94-41ba-8782-0a820814a41e","Type":"ContainerDied","Data":"f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6"} Nov 01 21:52:57.201188 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.201150 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl" event={"ID":"a924e870-cc94-41ba-8782-0a820814a41e","Type":"ContainerDied","Data":"b14eda4a7628b10ef792a79978c5966e675de18c4b14dd7adceaea6831aeef27"} Nov 01 21:52:57.208500 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.208479 5444 scope.go:122] "RemoveContainer" containerID="11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59" Nov 01 21:52:57.208814 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:52:57.208790 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59\": container with ID starting with 11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59 not found: ID does not exist" containerID="11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59" Nov 01 21:52:57.208927 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.208819 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59"} err="failed to get container status \"11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59\": rpc error: code = NotFound desc = could not find container \"11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59\": container with ID starting with 11b272e7d3425f5152cbaa5fdf2b1d8b8bfcbb2395e35f15c71485cac1114b59 not found: ID does not exist" Nov 01 21:52:57.208927 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.208836 5444 scope.go:122] "RemoveContainer" containerID="f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6" Nov 01 21:52:57.216036 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.216020 5444 scope.go:122] "RemoveContainer" containerID="f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6" Nov 01 21:52:57.216308 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:52:57.216292 5444 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6\": container with ID starting with f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6 not found: ID does not exist" containerID="f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6" Nov 01 21:52:57.216391 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.216314 5444 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6"} err="failed to get container status \"f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6\": rpc error: code = NotFound desc = could not find container \"f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6\": container with ID starting with f61b2fce418680e2d431b23b4bf5a5e73b02ee7eff0fcac33253820bec57a3a6 not found: ID does not exist" Nov 01 21:52:57.360534 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.360494 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl"] Nov 01 21:52:57.412178 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.411510 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-jdqnl"] Nov 01 21:52:57.471412 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.471385 5444 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5"] Nov 01 21:52:57.473052 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:57.473021 5444 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8n8s5"] Nov 01 21:52:59.156205 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:59.156172 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" path="/var/lib/kubelet/pods/1ab2ac29-924b-4cf1-bea6-cc72a48dd052/volumes" Nov 01 21:52:59.156606 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:52:59.156592 5444 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a924e870-cc94-41ba-8782-0a820814a41e" path="/var/lib/kubelet/pods/a924e870-cc94-41ba-8782-0a820814a41e/volumes" Nov 01 21:53:00.033823 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:53:00.033791 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:53:30.033931 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:53:30.033889 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:54:00.034262 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:54:00.034218 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:54:30.034406 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:54:30.034306 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:54:52.294450 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294412 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f"] Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294743 5444 cpu_manager.go:414] "containerMap: removing container" podUID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" containerName="maas-api" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294758 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" containerName="maas-api" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294773 5444 cpu_manager.go:414] "containerMap: removing container" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294779 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294789 5444 cpu_manager.go:414] "containerMap: removing container" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294794 5444 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294844 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b1cc77-92f0-4160-8f97-27d1e736ef46" containerName="maas-api" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294853 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="a924e870-cc94-41ba-8782-0a820814a41e" containerName="istio-proxy" Nov 01 21:54:52.294932 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.294860 5444 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab2ac29-924b-4cf1-bea6-cc72a48dd052" containerName="istio-proxy" Nov 01 21:54:52.297538 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.297522 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.343949 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.343924 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p"] Nov 01 21:54:52.346852 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.346834 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380431 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380405 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380540 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380439 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380540 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380456 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.380540 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380510 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380744 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380579 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380744 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380696 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blxw\" (UniqueName: \"kubernetes.io/projected/d8a243af-5f62-4e81-b03d-aa7214d6cee7-kube-api-access-8blxw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380744 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380735 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.380900 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380764 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.380900 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380798 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.380900 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380822 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.380900 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380854 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.380900 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380889 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.381086 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380916 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.381086 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380939 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.381086 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.380956 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.381086 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.381047 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.381086 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.381066 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.381258 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.381087 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngbb\" (UniqueName: \"kubernetes.io/projected/310b98cf-76eb-43e3-a6d5-bdad3590ba44-kube-api-access-dngbb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.402527 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.402503 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f"] Nov 01 21:54:52.452903 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.452868 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p"] Nov 01 21:54:52.482274 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482245 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blxw\" (UniqueName: \"kubernetes.io/projected/d8a243af-5f62-4e81-b03d-aa7214d6cee7-kube-api-access-8blxw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.482274 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482275 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482428 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482294 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482428 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482310 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.482428 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482360 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482428 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482404 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482435 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482469 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482500 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482524 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482565 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482590 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.482661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482624 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngbb\" (UniqueName: \"kubernetes.io/projected/310b98cf-76eb-43e3-a6d5-bdad3590ba44-kube-api-access-dngbb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482670 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482714 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482738 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482780 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482786 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482804 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482792 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482828 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482850 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.483042 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.482916 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.483561 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.483243 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483561 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.483386 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483561 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.483521 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.483561 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.483552 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.483800 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.483552 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.485265 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.485242 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.485407 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.485391 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.485471 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.485449 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.485512 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.485465 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.538398 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.538372 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310b98cf-76eb-43e3-a6d5-bdad3590ba44-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.538517 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.538376 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blxw\" (UniqueName: \"kubernetes.io/projected/d8a243af-5f62-4e81-b03d-aa7214d6cee7-kube-api-access-8blxw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.538588 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.538564 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngbb\" (UniqueName: \"kubernetes.io/projected/310b98cf-76eb-43e3-a6d5-bdad3590ba44-kube-api-access-dngbb\") pod \"maas-default-gateway-openshift-default-58847f9fc5-88m6f\" (UID: \"310b98cf-76eb-43e3-a6d5-bdad3590ba44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.555815 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.555761 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a243af-5f62-4e81-b03d-aa7214d6cee7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-4m95p\" (UID: \"d8a243af-5f62-4e81-b03d-aa7214d6cee7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:52.608918 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.608893 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:52.658958 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:52.658931 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:53.031174 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.031144 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f"] Nov 01 21:54:53.034220 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:54:53.034194 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310b98cf_76eb_43e3_a6d5_bdad3590ba44.slice/crio-17cc12c9dc4fbc6fcb53db5f50b602dc86522a9b0924d46a06cf2d067e8f6530 WatchSource:0}: Error finding container 17cc12c9dc4fbc6fcb53db5f50b602dc86522a9b0924d46a06cf2d067e8f6530: Status 404 returned error can't find the container with id 17cc12c9dc4fbc6fcb53db5f50b602dc86522a9b0924d46a06cf2d067e8f6530 Nov 01 21:54:53.036277 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.036247 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:54:53.036357 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.036319 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:54:53.036357 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.036348 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:54:53.086983 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.086886 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p"] Nov 01 21:54:53.090511 ip-10-0-138-177 kubenswrapper[5444]: W1101 21:54:53.090487 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a243af_5f62_4e81_b03d_aa7214d6cee7.slice/crio-739fc94b635cce84d605fb7217e6cf43b4e8eb0f221a17468f07386b182964fb WatchSource:0}: Error finding container 739fc94b635cce84d605fb7217e6cf43b4e8eb0f221a17468f07386b182964fb: Status 404 returned error can't find the container with id 739fc94b635cce84d605fb7217e6cf43b4e8eb0f221a17468f07386b182964fb Nov 01 21:54:53.092506 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.092479 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:54:53.092567 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.092538 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:54:53.092615 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.092567 5444 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 21:54:53.656605 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.656566 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" event={"ID":"d8a243af-5f62-4e81-b03d-aa7214d6cee7","Type":"ContainerStarted","Data":"9ccf62f406184d23d0cdc9e2e5fdd1402cafdd589e44e64e518906337c4cd682"} Nov 01 21:54:53.657483 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.657458 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" event={"ID":"d8a243af-5f62-4e81-b03d-aa7214d6cee7","Type":"ContainerStarted","Data":"739fc94b635cce84d605fb7217e6cf43b4e8eb0f221a17468f07386b182964fb"} Nov 01 21:54:53.659293 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.659271 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" event={"ID":"310b98cf-76eb-43e3-a6d5-bdad3590ba44","Type":"ContainerStarted","Data":"bcfa8fd8e8b203e5647011a0cac09e3077afd161d9ceae69cc90b599738e8dcb"} Nov 01 21:54:53.659703 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.659684 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" event={"ID":"310b98cf-76eb-43e3-a6d5-bdad3590ba44","Type":"ContainerStarted","Data":"17cc12c9dc4fbc6fcb53db5f50b602dc86522a9b0924d46a06cf2d067e8f6530"} Nov 01 21:54:53.660352 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.660336 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:54:53.774661 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:53.771998 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podStartSLOduration=1.771976958 podStartE2EDuration="1.771976958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:53.771093261 +0000 UTC m=+6694.715931130" watchObservedRunningTime="2026-11-01 21:54:53.771976958 +0000 UTC m=+6694.716814821" Nov 01 21:54:54.609723 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:54.609695 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:54:54.610777 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:54.610756 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:54:54.610840 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:54.610802 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:54:54.673671 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:54.670749 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:54:54.673671 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:54.670803 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:54:55.610327 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:55.610283 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:54:55.610518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:55.610353 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:54:55.663043 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:55.663004 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:54:55.663317 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:55.663285 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:54:56.609574 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:56.609536 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:54:56.609997 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:56.609606 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:54:56.661141 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:56.661002 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:54:56.661141 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:56.661081 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:54:57.609422 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:57.609384 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:54:57.609715 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:57.609461 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:54:57.662665 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:57.660930 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:54:57.662665 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:57.660985 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:54:58.609846 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:58.609805 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:54:58.610331 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:58.609861 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:54:58.664660 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:58.660966 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:54:58.664660 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:58.661022 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:54:59.609781 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:59.609745 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:54:59.609946 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:59.609810 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:54:59.659463 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:59.659431 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:54:59.659592 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:54:59.659485 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:55:00.034354 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:55:00.034314 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:55:00.609604 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:00.609566 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 01 21:55:00.609817 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:00.609631 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 01 21:55:00.660323 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:00.660278 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 01 21:55:00.660689 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:00.660351 5444 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 01 21:55:02.490518 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.490480 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:55:02.491619 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.491590 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:55:02.492433 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.492415 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" Nov 01 21:55:02.536126 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.536095 5444 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:55:02.536503 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.536470 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:55:02.537199 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.537182 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" Nov 01 21:55:02.654977 ip-10-0-138-177 kubenswrapper[5444]: I1101 21:55:02.654936 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podStartSLOduration=10.654925361 podStartE2EDuration="10.654925361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:53.891747275 +0000 UTC m=+6694.836585144" watchObservedRunningTime="2026-11-01 21:55:02.654925361 +0000 UTC m=+6703.599763223" Nov 01 21:55:30.033891 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:55:30.033855 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:56:00.035372 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:56:00.035295 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:56:30.034714 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:56:30.034675 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:57:00.035221 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:57:00.035188 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:57:30.034126 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:57:30.034080 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:58:00.034511 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:58:00.034465 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:58:30.034204 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:58:30.034160 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:59:00.034292 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:59:00.034215 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 21:59:30.034537 ip-10-0-138-177 kubenswrapper[5444]: E1101 21:59:30.034438 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:00:00.034783 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:00:00.034746 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:00:30.034771 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:00:30.034722 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:01:00.034860 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:01:00.034821 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:01:18.552812 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:01:18.552765 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:01:18.553466 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:01:18.553424 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:18.579780 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:01:18.579735 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:01:18.580009 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:01:18.579981 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:01:30.034381 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:01:30.034349 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:02:00.035326 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:02:00.035286 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:02:30.035809 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:02:30.035768 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:03:00.034235 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:03:00.034196 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:03:03.536806 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:03:03.536764 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:03.537209 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:03:03.536835 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:03.580661 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:03:03.579741 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:03.580661 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:03:03.579803 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:03:30.034076 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:03:30.034039 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:03:36.580796 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:03:36.580760 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:03:36.581614 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:03:36.581570 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:04:00.038597 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:04:00.038560 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:04:30.042258 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:04:30.042155 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:04:51.583860 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:04:51.583758 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gg89h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:04:51.584613 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:04:51.584576 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gg89h" podUID="d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:00.033915 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:05:00.033880 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:05:30.034549 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:05:30.034510 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:05:33.535808 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:05:33.535767 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-88m6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:33.536352 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:05:33.535836 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-88m6f" podUID="310b98cf-76eb-43e3-a6d5-bdad3590ba44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:05:33.580028 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:05:33.579900 5444 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-4m95p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:05:33.580028 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:05:33.579966 5444 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-4m95p" podUID="d8a243af-5f62-4e81-b03d-aa7214d6cee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:06:00.034394 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:06:00.034356 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:06:30.035775 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:06:30.035735 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:06:54.774427 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:06:54.774393 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qh45n_88630e47-4588-44a1-94ac-f39973ad1406/manager/0.log" Nov 01 22:06:57.293610 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:06:57.293576 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8595f64f4d-grjkl_73f82cfd-0abd-4086-8577-d059fa8d9d82/maas-api/0.log" Nov 01 22:07:00.034155 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:07:00.034123 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:07:01.274470 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:01.274439 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-5nrpg_2f54063d-c15d-4866-988b-a38a0a839dad/postgres/0.log" Nov 01 22:07:04.023457 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:04.023428 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vnw68_1c283462-a39d-406e-be48-ad3c33a27857/kuadrant-console-plugin/0.log" Nov 01 22:07:06.921199 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:06.921165 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hlppm_6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31/discovery/0.log" Nov 01 22:07:07.487309 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:07.487278 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c84dcdb75-q8bgn_081918c5-3f3d-4d01-aafe-b8f8ce741bfd/kube-auth-proxy/0.log" Nov 01 22:07:07.705623 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:07.705594 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-4m95p_d8a243af-5f62-4e81-b03d-aa7214d6cee7/istio-proxy/0.log" Nov 01 22:07:08.185925 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:08.185901 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-88m6f_310b98cf-76eb-43e3-a6d5-bdad3590ba44/istio-proxy/0.log" Nov 01 22:07:08.633307 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:08.633279 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-gg89h_d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4/istio-proxy/0.log" Nov 01 22:07:30.034226 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:07:30.034191 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:07:33.536440 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:33.536412 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tn5w8_ffffa9d9-1fea-40cb-85c6-1a1edb49470d/global-pull-secret-syncer/0.log" Nov 01 22:07:35.890082 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:35.890052 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c5z7h_b2d65d12-0f39-4bb4-9ced-cce27246077c/konnectivity-agent/0.log" Nov 01 22:07:35.993343 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:35.993314 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qh45n_88630e47-4588-44a1-94ac-f39973ad1406/manager/0.log" Nov 01 22:07:37.107385 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:37.107360 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-177.us-west-1.compute.internal_6f167604710ff032b6c087eabbd79a27/haproxy/0.log" Nov 01 22:07:41.532876 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:41.532844 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qh45n_88630e47-4588-44a1-94ac-f39973ad1406/manager/0.log" Nov 01 22:07:42.590432 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.590406 5444 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc"] Nov 01 22:07:42.593494 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.593477 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.620044 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.620016 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-sys\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.620168 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.620048 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-podres\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.620168 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.620065 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-lib-modules\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.620300 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.620221 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwgb\" (UniqueName: \"kubernetes.io/projected/6a59027f-6349-45c9-92c7-f90f3c4b8558-kube-api-access-bvwgb\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.620300 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.620259 5444 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-proc\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.645265 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.645239 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h9v88\"/\"openshift-service-ca.crt\"" Nov 01 22:07:42.694790 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.694765 5444 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h9v88\"/\"default-dockercfg-rbhbt\"" Nov 01 22:07:42.694908 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.694793 5444 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h9v88\"/\"kube-root-ca.crt\"" Nov 01 22:07:42.695855 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.695834 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc"] Nov 01 22:07:42.720902 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.720877 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-podres\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.720990 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.720908 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-lib-modules\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.720990 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.720964 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwgb\" (UniqueName: \"kubernetes.io/projected/6a59027f-6349-45c9-92c7-f90f3c4b8558-kube-api-access-bvwgb\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.721065 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.720988 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-proc\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.721065 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.721004 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-podres\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.721065 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.721057 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-proc\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.721164 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.721076 5444 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-sys\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.721164 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.721104 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-sys\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.721164 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.721078 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a59027f-6349-45c9-92c7-f90f3c4b8558-lib-modules\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.777317 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.777288 5444 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwgb\" (UniqueName: \"kubernetes.io/projected/6a59027f-6349-45c9-92c7-f90f3c4b8558-kube-api-access-bvwgb\") pod \"perf-node-gather-daemonset-mc8qc\" (UID: \"6a59027f-6349-45c9-92c7-f90f3c4b8558\") " pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:42.905179 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:42.905130 5444 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:43.269541 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:43.269511 5444 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc"] Nov 01 22:07:43.272744 ip-10-0-138-177 kubenswrapper[5444]: W1101 22:07:43.272710 5444 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a59027f_6349_45c9_92c7_f90f3c4b8558.slice/crio-b8deb2387cb675c1a1c1d69791ff695c190e3039041e64a00729caa464d180c3 WatchSource:0}: Error finding container b8deb2387cb675c1a1c1d69791ff695c190e3039041e64a00729caa464d180c3: Status 404 returned error can't find the container with id b8deb2387cb675c1a1c1d69791ff695c190e3039041e64a00729caa464d180c3 Nov 01 22:07:43.821493 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:43.821471 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vnw68_1c283462-a39d-406e-be48-ad3c33a27857/kuadrant-console-plugin/0.log" Nov 01 22:07:44.135560 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:44.135480 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" event={"ID":"6a59027f-6349-45c9-92c7-f90f3c4b8558","Type":"ContainerStarted","Data":"60725604e2a9047fee9fdebefe3b12e9ed621a1099b256522e601cb6af039ed2"} Nov 01 22:07:44.135560 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:44.135547 5444 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" event={"ID":"6a59027f-6349-45c9-92c7-f90f3c4b8558","Type":"ContainerStarted","Data":"b8deb2387cb675c1a1c1d69791ff695c190e3039041e64a00729caa464d180c3"} Nov 01 22:07:44.135794 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:44.135580 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:44.247079 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:44.247037 5444 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" podStartSLOduration=2.247023425 podStartE2EDuration="2.247023425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:44.24648959 +0000 UTC m=+7465.191327456" watchObservedRunningTime="2026-11-01 22:07:44.247023425 +0000 UTC m=+7465.191861272" Nov 01 22:07:46.749689 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:46.749656 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qh45n_88630e47-4588-44a1-94ac-f39973ad1406/manager/0.log" Nov 01 22:07:50.148843 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:50.148773 5444 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h9v88/perf-node-gather-daemonset-mc8qc" Nov 01 22:07:53.146018 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:53.145991 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-z6p67_78496de5-ccaa-448e-8683-52ea3c9e5de1/monitoring-plugin/0.log" Nov 01 22:07:54.823221 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:54.823185 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ll458_af45e616-0778-4869-9066-7acad174e56a/node-exporter/0.log" Nov 01 22:07:54.898577 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:54.898547 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hlppm_6d9f6b66-47d6-45b9-bd5d-4d36a72a6c31/discovery/0.log" Nov 01 22:07:55.214535 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:55.214459 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c84dcdb75-q8bgn_081918c5-3f3d-4d01-aafe-b8f8ce741bfd/kube-auth-proxy/0.log" Nov 01 22:07:55.238538 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:55.238516 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ll458_af45e616-0778-4869-9066-7acad174e56a/kube-rbac-proxy/0.log" Nov 01 22:07:55.425295 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:55.425275 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-4m95p_d8a243af-5f62-4e81-b03d-aa7214d6cee7/istio-proxy/0.log" Nov 01 22:07:55.657422 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:55.657397 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ll458_af45e616-0778-4869-9066-7acad174e56a/init-textfile/0.log" Nov 01 22:07:55.933341 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:55.933266 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-88m6f_310b98cf-76eb-43e3-a6d5-bdad3590ba44/istio-proxy/0.log" Nov 01 22:07:56.358741 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:56.358715 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-gg89h_d71d93d9-13cf-4a3d-afb4-3fdc6efc29a4/istio-proxy/0.log" Nov 01 22:07:57.330889 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:57.330857 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wfm9j_3eb7e9a5-c890-47bf-994a-db739d7e2817/kube-rbac-proxy-main/0.log" Nov 01 22:07:57.746892 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:57.746827 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wfm9j_3eb7e9a5-c890-47bf-994a-db739d7e2817/kube-rbac-proxy-self/0.log" Nov 01 22:07:58.162775 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:07:58.162750 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wfm9j_3eb7e9a5-c890-47bf-994a-db739d7e2817/openshift-state-metrics/0.log" Nov 01 22:08:00.034359 ip-10-0-138-177 kubenswrapper[5444]: E1101 22:08:00.034326 5444 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hz4mt" containerName="extractor" Nov 01 22:08:03.274198 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:03.274167 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vnw68_1c283462-a39d-406e-be48-ad3c33a27857/kuadrant-console-plugin/0.log" Nov 01 22:08:04.038562 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:04.038529 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5867784ff5-9s6fg_78d12c00-e8a7-4ac4-af79-b618b391618e/thanos-query/0.log" Nov 01 22:08:04.453499 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:04.453429 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5867784ff5-9s6fg_78d12c00-e8a7-4ac4-af79-b618b391618e/kube-rbac-proxy-web/0.log" Nov 01 22:08:04.869768 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:04.869738 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5867784ff5-9s6fg_78d12c00-e8a7-4ac4-af79-b618b391618e/kube-rbac-proxy/0.log" Nov 01 22:08:05.285334 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:05.285305 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5867784ff5-9s6fg_78d12c00-e8a7-4ac4-af79-b618b391618e/prom-label-proxy/0.log" Nov 01 22:08:05.701401 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:05.701333 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5867784ff5-9s6fg_78d12c00-e8a7-4ac4-af79-b618b391618e/kube-rbac-proxy-rules/0.log" Nov 01 22:08:06.116863 ip-10-0-138-177 kubenswrapper[5444]: I1101 22:08:06.116839 5444 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5867784ff5-9s6fg_78d12c00-e8a7-4ac4-af79-b618b391618e/kube-rbac-proxy-metrics/0.log"