Oct 31 01:03:30.003526 ip-10-0-141-86 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:03:30.411004 ip-10-0-141-86 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:30.411004 ip-10-0-141-86 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:03:30.411004 ip-10-0-141-86 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:30.411652 ip-10-0-141-86 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:30.417025 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.416942    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.417025 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417025    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417030    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417034    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417037    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417039    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417042    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417045    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417048    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417050    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417053    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417056    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417058    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417061    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417064    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417066    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417069    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417072    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417075    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417077    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.417098 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417086    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417089    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417092    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417094    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417097    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417099    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417102    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417105    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417108    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417111    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417113    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417116    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417119    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417121    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417124    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417126    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417129    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417131    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417134    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.417557 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417137    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417139    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417142    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417144    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417150    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417154    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417157    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417160    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417163    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417165    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417168    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417170    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417172    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417175    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417178    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417200    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417203    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417205    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417208    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.418113 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417211    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417213    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417216    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417218    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417221    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417223    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417226    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417229    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417231    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417234    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417236    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417238    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417241    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417244    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417247    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417251    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417254    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417257    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417259    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417262    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.418591 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417264    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417267    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417269    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417272    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417274    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417277    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417280    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417283    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417287    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417290    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417292    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417295    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417297    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417300    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417302    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417306    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417308    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417310    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417313    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417316    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.419066 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417318    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417321    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417323    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417325    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417328    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417330    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417333    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.417335    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418164    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418173    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418176    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418179    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418196    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418200    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418202    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418205    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418207    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418210    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418212    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418216    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.419558 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418219    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418223    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418226    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418229    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418232    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418234    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418237    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418240    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418242    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418245    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418248    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418250    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418253    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418258    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418261    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418263    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418266    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418268    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.420047 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418271    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418274    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418277    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418281    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418284    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418287    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418290    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418292    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418295    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418297    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418299    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418302    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418306    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418309    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418311    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418314    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418317    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418319    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418322    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.420528 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418324    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418327    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418330    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418332    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418335    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418337    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418339    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418342    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418344    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418347    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418349    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418352    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418354    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418357    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418361    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418364    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418366    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418368    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418371    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418373    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.421030 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418376    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418378    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418380    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418383    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418385    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418388    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418391    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418394    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418396    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418399    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418404    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418407    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418409    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418412    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418414    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418417    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418419    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418421    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418424    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.421537 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418426    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418428    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418431    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418433    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418435    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418438    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418440    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418442    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418446    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418448    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418451    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418453    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418456    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418459    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418463    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418467    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.418996    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419002    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419006    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.422042 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419010    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419013    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419017    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419019    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419022    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419025    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419027    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419030    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419032    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419035    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419038    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419040    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419043    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419045    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419048    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419051    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419053    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419056    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419058    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419061    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.422521 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419063    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419066    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419069    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419072    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419074    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419077    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419079    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419082    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419085    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419087    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419090    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419093    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419095    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419098    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419100    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419106    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419109    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419114    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419116    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.423017 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419118    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419121    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419123    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419126    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419128    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419131    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419133    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419136    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419138    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419141    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419143    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419145    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419148    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419151    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419154    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419156    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419158    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419161    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419164    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419166    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.423561 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419168    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419171    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419174    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419176    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419179    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419196    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419199    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419202    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419205    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419207    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419210    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419213    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419216    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419218    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419220    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419223    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419226    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419230    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419234    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.424054 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419237    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419239    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419242    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419245    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419247    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419250    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419252    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419255    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419258    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419261    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419263    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419266    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419268    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419271    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419275    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419281    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419283    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419286    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419288    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.424540 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419291    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419293    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419297    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419300    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.419303    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419392    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419399    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419405    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419410    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419414    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419418    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419422    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419427    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419430    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419433    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419437    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419440    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419443    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419446    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419451    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419454    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419457    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419460    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419464    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:03:30.424983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419467    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419470    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419473    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419476    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419482    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419485    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419488    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419492    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419495    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419498    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419501    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419504    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419507    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419512    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419515    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419518    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419520    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419524    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419527    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419531    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419534    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419537    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419540    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419543    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:03:30.425556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419555    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419558    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419561    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419566    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419569    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419573    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419576    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419580    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419583    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419585    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419588    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419592    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419597    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419601    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419604    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419607    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419610    2577 flags.go:64] FLAG: --help="false"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419613    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419616    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419620    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419622    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419626    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419630    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:03:30.426203 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419633    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419636    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419639    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419642    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419644    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419648    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419650    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419653    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419656    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419660    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419662    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419665    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419669    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419672    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419675    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419681    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419684    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419687    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419690    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419693    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419696    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419700    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419703    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419707    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419711    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:03:30.426787 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419715    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419718    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419721    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419724    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419727    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419729    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419733    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419736    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419743    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419747    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419750    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419753    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419756    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419758    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419762    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419765    2577 flags.go:64] FLAG: --port="10250"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419768    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419770    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0cd84a8000fadabfa"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419773    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419778    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419781    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419784    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419787    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419790    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419793    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419796    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:03:30.427403 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419800    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419803    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419807    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419810    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419813    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419816    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419819    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419823    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419825    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419828    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419832    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419835    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419838    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419841    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419844    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419847    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419850    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419853    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419856    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419859    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419864    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419867    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419870    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419874    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419877    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:03:30.428024 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419881    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419884    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419886    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419889    2577 flags.go:64] FLAG: --v="2"
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419894    2577 flags.go:64] FLAG: --version="false"
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419898    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419902    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.419906    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420067    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420073    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420076    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420080    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420083    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420088    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420090    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420093    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420096    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420098    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420101    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420103    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.428640 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420106    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420109    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420111    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420114    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420117    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420119    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420121    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420124    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420126    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420129    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420131    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420136    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420139    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420141    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420143    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420146    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420148    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420150    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420153    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.429140 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420155    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420159    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420162    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420164    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420167    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420169    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420172    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420174    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420176    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420179    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420198    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420201    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420203    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420206    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420209    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420212    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420214    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420217    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420219    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420221    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.429609 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420224    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420226    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420229    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420231    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420236    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420238    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420241    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420243    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420246    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420248    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420251    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420253    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420256    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420260    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420270    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420273    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420276    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420278    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.430077 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420285    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420289    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420292    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420295    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420298    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420301    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420303    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420306    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420309    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420312    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420315    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420317    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420320    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420322    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420325    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420327    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420330    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420334    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420336    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.430813 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420340    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420343    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420346    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420349    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420351    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420354    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420356    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420361    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420364    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420367    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420370    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420372    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420375    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420378    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420380    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420382    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.431522 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.420409    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420454    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420459    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420462    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420465    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420467    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420470    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420473    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420475    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420478    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420481    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420483    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420486    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420489    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420493    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420495    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420498    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420500    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420503    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420505    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.431960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420508    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420510    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420513    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420515    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420519    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420521    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420524    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420526    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420529    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420531    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420534    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420537    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420539    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420542    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420544    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420547    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420550    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420552    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420554    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.432511 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420557    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420560    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420562    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420565    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420567    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420570    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420572    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420576    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420579    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420581    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420584    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420586    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420589    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420591    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420594    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420596    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420598    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420602    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420604    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420607    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.432960 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420609    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420612    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420614    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420616    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420619    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420621    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420623    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420626    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420628    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420631    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420633    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420636    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420638    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420641    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420643    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420646    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420648    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420650    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420653    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.433453 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420657    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420661    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420665    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420668    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420671    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420674    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420677    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420679    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420681    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420684    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420687    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420689    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420691    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420694    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420696    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420698    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420701    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420703    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420706    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.433901 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420708    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420712    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420714    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420717    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420720    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420724    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420726    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420728    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420731    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.420734    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.427652    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.427668    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.428315    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.428321    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428376    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428383    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428387    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428392    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428395    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.434378 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428398    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428401    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428403    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428406    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428409    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428412    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428414    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428417    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428420    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428423    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428426    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428428    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428431    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428434    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428436    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428438    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428441    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428443    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428446    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428448    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.434827 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428451    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428454    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428456    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428459    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428462    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428465    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428467    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428470    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428473    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428489    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428493    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428497    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428500    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428503    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428506    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428508    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428511    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428514    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428518    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428521    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.435320 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428525    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428528    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428530    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428533    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428535    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428538    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428540    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428543    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428545    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428548    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428550    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428553    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428555    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428558    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428561    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428564    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428567    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428569    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428572    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.435805 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428575    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428578    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428580    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428585    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428588    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428590    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428592    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428595    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428598    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428600    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428602    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428605    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428607    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428610    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428612    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428614    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428617    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428619    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.436348 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428621    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428624    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428626    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428629    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428631    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428633    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428636    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428638    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428641    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428644    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428647    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428650    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428653    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428655    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428658    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428661    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428663    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428666    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428668    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428671    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.436836 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428673    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428676    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428678    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.428708    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428836    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428841    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428846    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428849    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428852    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428854    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428857    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428860    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428862    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428865    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428867    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.437318 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428870    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428872    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428875    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428878    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428880    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428883    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428885    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428889    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428891    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428894    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428896    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428899    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428901    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428903    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428906    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428908    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428910    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428913    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428915    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428918    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.437673 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428920    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428922    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428925    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428927    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428929    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428932    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428934    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428936    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428939    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428942    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428944    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428947    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428949    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428952    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428954    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428957    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428959    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428962    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428964    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428967    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.438136 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428970    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428973    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428975    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428979    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428982    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428985    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428987    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428991    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428993    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428996    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.428998    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429000    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429003    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429005    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429008    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429010    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429013    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429015    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429017    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.438617 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429020    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429023    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429025    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429028    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429030    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429033    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429035    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429037    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429040    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429042    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429045    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429047    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429050    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429052    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429059    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429062    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429064    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429066    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429069    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429071    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.439056 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429074    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429076    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429078    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429080    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429083    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429086    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429089    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429091    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429093    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429096    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429099    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429101    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429104    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429106    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429109    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429111    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.439599 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.429134    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429177    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429199    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429205    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429210    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429215    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429217    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429221    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429223    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429226    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429228    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429232    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429235    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429237    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429240    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429242    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429245    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429247    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429250    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429252    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:30.439982 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429255    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429257    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429260    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429262    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429264    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429267    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429269    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429272    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429274    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429277    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429280    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429283    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429285    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429288    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429291    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429293    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429296    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429298    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429300    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:30.440469 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429304    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429309    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429312    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429315    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429318    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429321    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429324    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429326    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429329    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429331    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429333    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429336    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429338    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429340    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429343    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429345    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429347    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429350    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429352    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429354    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:30.440954 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429357    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429359    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429362    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429364    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429367    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429369    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429372    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429374    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429376    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429379    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429381    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429384    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429387    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429390    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429392    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429394    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429397    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429399    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429406    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429409    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:30.441432 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429412    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429414    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429417    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429419    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429421    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429424    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429426    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429429    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429431    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429433    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429436    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429438    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429441    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429445    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429449    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429454    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429456    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429459    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429462    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:30.441905 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429464    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429467    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429469    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429472    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429474    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429477    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429480    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429482    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:30.429484    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.429538    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.433635    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.434604    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.434692    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.434721    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.436879    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:30.442375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.438053    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:30.447532 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.447517    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:03:30.450689 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.450674    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 01:03:30.451610 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.451580    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:03:30.453926 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.453905    2577 fs.go:135] Filesystem UUIDs: map[21439fdd-9ff9-460d-8fda-3306979c2c32:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f32ddd08-39f2-4ca2-9d5b-2c0526f294af:/dev/nvme0n1p3]
Oct 31 01:03:30.453988 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.453925    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 01:03:30.459525 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.459407    2577 manager.go:217] Machine: {Timestamp:2026-10-31 01:03:30.458075477 +0000 UTC m=+0.330675887 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100466 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2500774b2f03d360526cd57e9f7270 SystemUUID:ec250077-4b2f-03d3-6052-6cd57e9f7270 BootID:93c51d36-cdef-4404-836e-79b83c32e53e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:86:98:70:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:86:98:70:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:8d:67:d1:41:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:03:30.459525 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.459528    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:03:30.459632 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.459609    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:03:30.462250 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.462229    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:03:30.462392 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.462251    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-86.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:03:30.462439 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.462395    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:03:30.462439 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.462403    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:03:30.462494 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.462439    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:30.462494 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.462457    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:30.463108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.463097    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:03:30.466656 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.466641    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:03:30.466795 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.466785    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:03:30.467394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.467384    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:03:30.467434 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.467399    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:03:30.467434 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.467409    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:03:30.467434 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.467419    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:03:30.467434 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.467428    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:03:30.467778 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.467757    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:30.471854 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.471822    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:03:30.476411 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.476397    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:30.477166 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477154    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:03:30.477220 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477210    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:03:30.477394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477384    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:03:30.477426 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477401    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:03:30.477426 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477407    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:03:30.477426 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477413    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:03:30.477426 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477419    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:03:30.477529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.477427    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:03:30.478141 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478133    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:03:30.478171 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478143    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:03:30.478171 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478150    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:03:30.478171 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478156    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:03:30.478171 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478165    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:03:30.478345 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478173    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:03:30.478752 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478743    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:03:30.478781 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.478753    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:03:30.482506 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.482493    2577 server.go:1263] "Started kubelet"
Oct 31 01:03:30.482603 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.482579    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:03:30.482658 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.482611    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:03:30.482713 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.482703    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:03:30.483324 ip-10-0-141-86 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:03:30.483698 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.483677    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:03:30.485397 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.485293    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:03:30.488866 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.488846    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 01:03:30.490089 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.490074    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:30.490465 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.490448    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:03:30.490549 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.490538    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:30.491141 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491124    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:03:30.491243 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491148    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:03:30.491243 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491162    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:03:30.491413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491262    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:03:30.491413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491271    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:03:30.491413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491297    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:03:30.491413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491308    2577 factory.go:55] Registering systemd factory
Oct 31 01:03:30.491413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491315    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:03:30.491691 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491675    2577 factory.go:153] Registering CRI-O factory
Oct 31 01:03:30.491691 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491691    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:03:30.491809 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491706    2577 factory.go:103] Registering Raw factory
Oct 31 01:03:30.491809 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.491716    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:03:30.492737 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.492715    2577 manager.go:319] Starting recovery of all containers
Oct 31 01:03:30.504019 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.503992    2577 manager.go:324] Recovery completed
Oct 31 01:03:30.508948 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.508927    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.511045 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511024    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.511128 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511060    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.511128 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511072    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.511711 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511697    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:03:30.511761 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511711    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:03:30.511761 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511713    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.511761 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.511734    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:03:30.512477 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.512459    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.512557 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.512494    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.512557 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.512505    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.514009 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.513995    2577 policy_none.go:50] "Start"
Oct 31 01:03:30.514073 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.514012    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:03:30.514073 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.514023    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:03:30.516074 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.516063    2577 policy_none.go:44] "Start"
Oct 31 01:03:30.556164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.555974    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583018    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.583060    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583073    2577 server.go:87] "Starting device plugin registration server"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583359    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583370    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583527    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583683    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.583691    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.584138    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:03:30.597003 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.584204    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:30.622808 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.622773    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-86.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 01:03:30.622808 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.622781    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-86.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 01:03:30.623163 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.623140    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:03:30.623975 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.622794    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-86.us-west-1.compute.internal.18e377a11cef2b77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:30.482465655 +0000 UTC m=+0.355066065,LastTimestamp:2026-10-31 01:03:30.482465655 +0000 UTC m=+0.355066065,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.us-west-1.compute.internal,}"
Oct 31 01:03:30.624641 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.624623    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:03:30.624718 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.624647    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:03:30.624718 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.624667    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:03:30.624718 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.624702    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 01:03:30.684215 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.684125    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.685938 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.685917    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.686046 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.685951    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.686046 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.685968    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.686046 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.685993    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.724821 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.724786    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"]
Oct 31 01:03:30.724965 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.724853    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.726109 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.726088    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.726259 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.726122    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.726259 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.726138    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.728017 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728004    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.728207 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.728256 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728239    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.728776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728761    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.728842 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728793    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.728842 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728803    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.728842 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728765    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.728938 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728848    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.728938 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.728859    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.730573 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.730558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.730629 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.730589    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:30.731268 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.731251    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:30.731367 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.731286    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:30.731367 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.731301    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:30.758349 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.758326    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.758427 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.758364    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:03:30.758427 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.758410    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9jwh7"
Oct 31 01:03:30.761777 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.761762    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.776499 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.776476    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 01:03:30.776658 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.776636    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 01:03:30.793693 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.793664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.793793 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.793721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.793793 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.793743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fafe7c61d6af085c60e65c5248ab0754-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"fafe7c61d6af085c60e65c5248ab0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.825428 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.825374    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-86.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 01:03:30.825520 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.825366    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-86.us-west-1.compute.internal.18e377a11ea3441e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-86.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:30.511045662 +0000 UTC m=+0.383646072,LastTimestamp:2026-10-31 01:03:30.511045662 +0000 UTC m=+0.383646072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.us-west-1.compute.internal,}"
Oct 31 01:03:30.844367 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.844341    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:30.844981 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.844963    2577 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-86.us-west-1.compute.internal\" not found" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.894661 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.894626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.894661 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.894659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.894847 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.894678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fafe7c61d6af085c60e65c5248ab0754-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"fafe7c61d6af085c60e65c5248ab0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.894847 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.894706    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fafe7c61d6af085c60e65c5248ab0754-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"fafe7c61d6af085c60e65c5248ab0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.894847 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.894718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.894847 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:30.894718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:30.925804 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:30.925773    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-86.us-west-1.compute.internal\" not found" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:31.045996 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.045912    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:31.046905 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.046884    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:31.047018 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.046916    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:31.047018 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.046926    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:31.047018 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.046954    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:31.061016 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.060994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:31.064309 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.064289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:31.101017 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.100992    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:31.101102 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.101019    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-86.us-west-1.compute.internal\": node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.217425 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.217390    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.318163 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.318089    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.419095 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.419062    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.434232 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.434205    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 01:03:31.434425 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.434397    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 01:03:31.434505 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.434417    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a53752eb2738248268d838fbfc908485-8edbb764ce6e4113.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-86.us-west-1.compute.internal.18e377a11ea3b973\": read tcp 10.0.141.86:57634->3.151.241.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-86.us-west-1.compute.internal.18e377a11ea3b973  default   5310 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-86.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:30 +0000 UTC,LastTimestamp:2026-10-31 01:03:30.685973117 +0000 UTC m=+0.558573527,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.us-west-1.compute.internal,}"
Oct 31 01:03:31.520110 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.520077    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.620786 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.620709    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.704373 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.704340    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:31.705421 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.705403    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:31.705529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.705444    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:31.705529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.705457    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:31.721078 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.721059    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.763402 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.763365    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:58:31 +0000 UTC" deadline="2028-04-21 00:13:13.24612539 +0000 UTC"
Oct 31 01:03:31.763402 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.763395    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12911h9m41.482733072s"
Oct 31 01:03:31.821279 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.821254    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.922154 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:31.922125    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:31.996352 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:31.996324    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:32.022646 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:32.022582    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:03:33.202214 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.202168    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:33.280164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.280100    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:33.340377 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.340348    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:33.342296 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.342281    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:03:33.371826 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:33.371791    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafe7c61d6af085c60e65c5248ab0754.slice/crio-234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e WatchSource:0}: Error finding container 234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e: Status 404 returned error can't find the container with id 234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e
Oct 31 01:03:33.372191 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:33.372170    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed86943c8c1fe4dccb7ac2c7a181775.slice/crio-53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce WatchSource:0}: Error finding container 53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce: Status 404 returned error can't find the container with id 53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce
Oct 31 01:03:33.376921 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.376904    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:03:33.440008 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.439971    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:33.440171 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.440016    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:33.558035 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.557941    2577 apiserver.go:52] "Watching apiserver"
Oct 31 01:03:33.580128 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.580093    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:33.660742 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.660705    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:03:33.661053 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.661035    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-rrnxg","kube-system/konnectivity-agent-j6h6v","openshift-cluster-node-tuning-operator/tuned-psw7z","openshift-image-registry/node-ca-vkb7h","openshift-multus/multus-additional-cni-plugins-9lpwg","openshift-multus/network-metrics-daemon-rdsvq","kube-system/global-pull-secret-syncer-pf5kd","openshift-multus/multus-68tmm","openshift-network-diagnostics/network-check-target-fczdh","openshift-ovn-kubernetes/ovnkube-node-rm9cp","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg","openshift-dns/node-resolver-jjdkz"]
Oct 31 01:03:33.664718 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.664701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:33.664810 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.664793    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:33.666749 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.666735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.668889 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.668870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.670919 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.670900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.672945 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.672927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.675247 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.675227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.677735 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.677718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.679946 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.679928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.682111 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.682096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.684628 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.684610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:33.684748 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.684685    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:33.687005 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.686984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.689318 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.689302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.689422 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.689364    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:33.695352 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-kubernetes\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.695430 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-tmp\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.695430 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.695430 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-socket-dir-parent\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.695590 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46tq\" (UniqueName: \"kubernetes.io/projected/e9562f13-b930-4947-9aac-9379535e8fe6-kube-api-access-p46tq\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.695590 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjhl\" (UniqueName: \"kubernetes.io/projected/2a09fe47-166b-4fa0-9982-afffd73320b8-kube-api-access-2xjhl\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.695590 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.695590 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn884\" (UniqueName: \"kubernetes.io/projected/de7aadea-e0a6-4086-ae7a-a8f7794ad264-kube-api-access-gn884\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.695590 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-tuned\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.695859 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g25n\" (UniqueName: \"kubernetes.io/projected/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-kube-api-access-2g25n\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:33.695859 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-system-cni-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.695859 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de7aadea-e0a6-4086-ae7a-a8f7794ad264-hosts-file\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.695859 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7aadea-e0a6-4086-ae7a-a8f7794ad264-tmp-dir\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.695859 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-multus-daemon-config\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.696063 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-conf\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.696063 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-run\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.696063 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.695982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-socket-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.696063 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-sys-fs\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.696063 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysconfig\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-registration-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa76e03-b876-4bc5-a0ec-aab3cced55f0-agent-certs\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-serviceca\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-cnibin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.696272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a09fe47-166b-4fa0-9982-afffd73320b8-host-slash\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa76e03-b876-4bc5-a0ec-aab3cced55f0-konnectivity-ca\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-k8s-cni-cncf-io\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a09fe47-166b-4fa0-9982-afffd73320b8-iptables-alerter-script\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-device-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lkz\" (UniqueName: \"kubernetes.io/projected/fb29d15c-7076-454a-a335-22e9b8097c4f-kube-api-access-f5lkz\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-lib-modules\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-host\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.696580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-cni-binary-copy\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697020 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-multus\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697020 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.697020 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.696982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-var-lib-kubelet\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.697149 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-host\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.697149 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-conf-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697149 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cnibin\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.697285 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.697285 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-systemd\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.697382 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-system-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697382 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-etc-kubernetes\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697476 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-modprobe-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.697476 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-kubelet\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697576 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc2r4\" (UniqueName: \"kubernetes.io/projected/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-kube-api-access-pc2r4\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.697751 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-multus-certs\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697751 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qs9\" (UniqueName: \"kubernetes.io/projected/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-kube-api-access-w2qs9\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.697871 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.697871 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-769kz\" (UniqueName: \"kubernetes.io/projected/7c3cee94-19dc-4e56-b7a2-256765111acb-kube-api-access-769kz\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.697871 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-netns\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697999 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-hostroot\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.697999 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.697999 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-sys\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.697999 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.697996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-bin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.718086 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.718033    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"fafe7c61d6af085c60e65c5248ab0754","Type":"ContainerStarted","Data":"234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e"}
Oct 31 01:03:33.718966 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.718943    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.719082 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.718966    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:03:33.719082 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.718967    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.719082 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.719043    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rnpg9\""
Oct 31 01:03:33.719341 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.719325    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"8ed86943c8c1fe4dccb7ac2c7a181775","Type":"ContainerStarted","Data":"53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce"}
Oct 31 01:03:33.720152 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.720139    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:03:33.720197 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.720149    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-55dd6\""
Oct 31 01:03:33.720966 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.720952    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:03:33.722286 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.722269    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.722377 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.722273    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.722527 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.722516    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9z86\""
Oct 31 01:03:33.724126 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.724110    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:03:33.724222 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.724205    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kk499\""
Oct 31 01:03:33.724255 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.724212    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.724255 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.724216    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.726905 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.726891    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:03:33.727346 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.727331    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:03:33.727404 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.727363    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.727404 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.727391    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:03:33.727487 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.727401    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-grw5x\""
Oct 31 01:03:33.727487 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.727421    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.729135 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.729123    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:03:33.729828 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.729815    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zzhf8\""
Oct 31 01:03:33.731776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.731762    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.731776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.731763    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.731867 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.731785    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g746\""
Oct 31 01:03:33.733504 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.733492    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42fft\""
Oct 31 01:03:33.733783 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.733768    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.733846 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.733795    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.733920 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.733910    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:03:33.780509 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.780487    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:03:33.798452 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-multus-daemon-config\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.798575 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.798575 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-script-lib\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.798575 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-conf\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.798575 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-run\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.798765 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-socket-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.798765 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-sys-fs\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.798765 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.798765 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-sys-fs\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.798765 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysconfig\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-conf\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-run\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-registration-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798831    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysconfig\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa76e03-b876-4bc5-a0ec-aab3cced55f0-agent-certs\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798834    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-socket-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-registration-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-serviceca\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.798905    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-systemd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.798949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-ovn\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.798991 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.798997    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:34.298950275 +0000 UTC m=+3.083083212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-cnibin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a09fe47-166b-4fa0-9982-afffd73320b8-host-slash\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a09fe47-166b-4fa0-9982-afffd73320b8-host-slash\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-cnibin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799130    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-multus-daemon-config\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-dbus\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-slash\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799195    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-bin\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-netd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799281    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-serviceca\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa76e03-b876-4bc5-a0ec-aab3cced55f0-konnectivity-ca\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-k8s-cni-cncf-io\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a09fe47-166b-4fa0-9982-afffd73320b8-iptables-alerter-script\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.799612 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-k8s-cni-cncf-io\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-device-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lkz\" (UniqueName: \"kubernetes.io/projected/fb29d15c-7076-454a-a335-22e9b8097c4f-kube-api-access-f5lkz\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-node-log\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799559    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-device-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-lib-modules\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-host\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-lib-modules\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-host\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-cni-binary-copy\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-multus\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmqv\" (UniqueName: \"kubernetes.io/projected/2c464f6d-9424-478f-8b7c-c64b564d816f-kube-api-access-jxmqv\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799847    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-multus\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa76e03-b876-4bc5-a0ec-aab3cced55f0-konnectivity-ca\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a09fe47-166b-4fa0-9982-afffd73320b8-iptables-alerter-script\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.800320 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-var-lib-kubelet\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-host\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-config\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.799997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-var-lib-kubelet\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-conf-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-host\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800123    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-cni-binary-copy\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c464f6d-9424-478f-8b7c-c64b564d816f-ovn-node-metrics-cert\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-conf-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cnibin\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-kubelet-config\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-systemd\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-systemd\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cnibin\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-system-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.800866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-etc-kubernetes\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-modprobe-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-kubelet\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-system-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2r4\" (UniqueName: \"kubernetes.io/projected/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-kube-api-access-pc2r4\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800868    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-etc-kubernetes\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800905    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-kubelet\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800912    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-modprobe-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-multus-certs\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qs9\" (UniqueName: \"kubernetes.io/projected/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-kube-api-access-w2qs9\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.800989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-multus-certs\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-var-lib-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-769kz\" (UniqueName: \"kubernetes.io/projected/7c3cee94-19dc-4e56-b7a2-256765111acb-kube-api-access-769kz\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801176    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-netns\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.801529 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801337    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-netns\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-hostroot\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801368    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801379    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-hostroot\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-kubelet\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-etc-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-sys\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-bin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-log-socket\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801605    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-sys\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-env-overrides\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-bin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-kubernetes\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-tmp\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802164 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-socket-dir-parent\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-socket-dir-parent\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801825    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-kubernetes\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46tq\" (UniqueName: \"kubernetes.io/projected/e9562f13-b930-4947-9aac-9379535e8fe6-kube-api-access-p46tq\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjhl\" (UniqueName: \"kubernetes.io/projected/2a09fe47-166b-4fa0-9982-afffd73320b8-kube-api-access-2xjhl\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.801942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn884\" (UniqueName: \"kubernetes.io/projected/de7aadea-e0a6-4086-ae7a-a8f7794ad264-kube-api-access-gn884\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-tuned\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g25n\" (UniqueName: \"kubernetes.io/projected/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-kube-api-access-2g25n\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-system-cni-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802231    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-system-cni-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de7aadea-e0a6-4086-ae7a-a8f7794ad264-hosts-file\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7aadea-e0a6-4086-ae7a-a8f7794ad264-tmp-dir\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-systemd-units\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802302    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de7aadea-e0a6-4086-ae7a-a8f7794ad264-hosts-file\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.802880 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-netns\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.803312 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.802549    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7aadea-e0a6-4086-ae7a-a8f7794ad264-tmp-dir\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.803385 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.803365    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa76e03-b876-4bc5-a0ec-aab3cced55f0-agent-certs\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.804020 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.804001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-tmp\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.804277 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.804261    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-tuned\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.814871 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.814816    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:03:33.814946 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.814888    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hc4nc"
Oct 31 01:03:33.819769 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.819746    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:03:33.819920 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.819748    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:03:33.819920 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.819749    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:03:33.819920 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.819791    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:03:33.819920 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.819804    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:03:33.820101 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.820024    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8rwd\""
Oct 31 01:03:33.820101 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.820056    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:03:33.858386 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.858351    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lkz\" (UniqueName: \"kubernetes.io/projected/fb29d15c-7076-454a-a335-22e9b8097c4f-kube-api-access-f5lkz\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.859168 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.859134    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2r4\" (UniqueName: \"kubernetes.io/projected/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-kube-api-access-pc2r4\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:33.859305 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.859277    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjhl\" (UniqueName: \"kubernetes.io/projected/2a09fe47-166b-4fa0-9982-afffd73320b8-kube-api-access-2xjhl\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:33.859305 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.859287    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qs9\" (UniqueName: \"kubernetes.io/projected/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-kube-api-access-w2qs9\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.859397 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.859338    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46tq\" (UniqueName: \"kubernetes.io/projected/e9562f13-b930-4947-9aac-9379535e8fe6-kube-api-access-p46tq\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:03:33.859636 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.859616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g25n\" (UniqueName: \"kubernetes.io/projected/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-kube-api-access-2g25n\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:33.860117 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.860095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-769kz\" (UniqueName: \"kubernetes.io/projected/7c3cee94-19dc-4e56-b7a2-256765111acb-kube-api-access-769kz\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.860254 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.860238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn884\" (UniqueName: \"kubernetes.io/projected/de7aadea-e0a6-4086-ae7a-a8f7794ad264-kube-api-access-gn884\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:33.903321 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:33.903321 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-systemd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-ovn\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-dbus\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-slash\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-bin\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903426    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-systemd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-ovn\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-netd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-dbus\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-slash\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-netd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-node-log\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903505    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-bin\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.903550 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-node-log\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmqv\" (UniqueName: \"kubernetes.io/projected/2c464f6d-9424-478f-8b7c-c64b564d816f-kube-api-access-jxmqv\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-config\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c464f6d-9424-478f-8b7c-c64b564d816f-ovn-node-metrics-cert\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-kubelet-config\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-var-lib-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-kubelet\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-etc-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903827    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-kubelet-config\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-kubelet\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.903834    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-log-socket\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-env-overrides\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904155 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903899    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-log-socket\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.903912    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:03:34.403890406 +0000 UTC m=+3.188023348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903836    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-etc-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903864    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-var-lib-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-systemd-units\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.903996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-netns\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-systemd-units\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-script-lib\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-netns\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-env-overrides\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.904918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.904698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-config\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.905433 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.905072    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-script-lib\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.908138 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.908115    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c464f6d-9424-478f-8b7c-c64b564d816f-ovn-node-metrics-cert\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.956970 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.956936    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:33.958272 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.958246    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:33.958272 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.958275    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:33.958453 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.958287    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:33.958453 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:33.958356    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:34.45833843 +0000 UTC m=+3.242471369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:33.960579 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.960553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmqv\" (UniqueName: \"kubernetes.io/projected/2c464f6d-9424-478f-8b7c-c64b564d816f-kube-api-access-jxmqv\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:33.981569 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.981547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:03:33.983635 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.983621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:33.989580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.989567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:03:33.993612 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:33.993584    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb29d15c_7076_454a_a335_22e9b8097c4f.slice/crio-c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997 WatchSource:0}: Error finding container c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997: Status 404 returned error can't find the container with id c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997
Oct 31 01:03:33.994398 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:33.994377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:03:33.994667 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:33.994632    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa76e03_b876_4bc5_a0ec_aab3cced55f0.slice/crio-bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c WatchSource:0}: Error finding container bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c: Status 404 returned error can't find the container with id bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c
Oct 31 01:03:33.998436 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:33.998414    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3cee94_19dc_4e56_b7a2_256765111acb.slice/crio-85516069c50b286e55c6f6d8c831420de807b4cbd34b20f452460cbb057e964d WatchSource:0}: Error finding container 85516069c50b286e55c6f6d8c831420de807b4cbd34b20f452460cbb057e964d: Status 404 returned error can't find the container with id 85516069c50b286e55c6f6d8c831420de807b4cbd34b20f452460cbb057e964d
Oct 31 01:03:34.004940 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:34.004917    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9773ac4_0a19_4f7c_a5f7_8a7d1cb38603.slice/crio-db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89 WatchSource:0}: Error finding container db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89: Status 404 returned error can't find the container with id db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89
Oct 31 01:03:34.013936 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.013918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68tmm"
Oct 31 01:03:34.016057 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.016036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:03:34.017461 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.017442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:03:34.020181 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.020166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:03:34.026495 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:34.026471    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9562f13_b930_4947_9aac_9379535e8fe6.slice/crio-73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7 WatchSource:0}: Error finding container 73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7: Status 404 returned error can't find the container with id 73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7
Oct 31 01:03:34.027289 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:34.027268    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7aadea_e0a6_4086_ae7a_a8f7794ad264.slice/crio-0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35 WatchSource:0}: Error finding container 0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35: Status 404 returned error can't find the container with id 0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35
Oct 31 01:03:34.031094 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:34.031024    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8405b7_4f76_495c_b9fc_04affdf8d1f8.slice/crio-297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20 WatchSource:0}: Error finding container 297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20: Status 404 returned error can't find the container with id 297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20
Oct 31 01:03:34.034028 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:34.034008    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a09fe47_166b_4fa0_9982_afffd73320b8.slice/crio-e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6 WatchSource:0}: Error finding container e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6: Status 404 returned error can't find the container with id e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6
Oct 31 01:03:34.039341 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.039323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:34.048991 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:03:34.048969    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c464f6d_9424_478f_8b7c_c64b564d816f.slice/crio-54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0 WatchSource:0}: Error finding container 54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0: Status 404 returned error can't find the container with id 54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0
Oct 31 01:03:34.306959 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.306926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:34.307149 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.307095    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:34.307213 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.307176    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:35.307155071 +0000 UTC m=+4.091288001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:34.407974 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.407932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:34.408152 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.408103    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:34.408225 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.408187    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:03:35.408165697 +0000 UTC m=+4.192298637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:34.508743 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.508699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:34.508932 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.508905    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:34.508932 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.508930    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:34.509019 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.508943    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:34.509019 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:34.508998    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:35.50898217 +0000 UTC m=+4.293115090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:34.723366 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.723276    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-psw7z" event={"ID":"7c3cee94-19dc-4e56-b7a2-256765111acb","Type":"ContainerStarted","Data":"85516069c50b286e55c6f6d8c831420de807b4cbd34b20f452460cbb057e964d"}
Oct 31 01:03:34.724773 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.724743    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0"}
Oct 31 01:03:34.726356 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.726324    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrnxg" event={"ID":"2a09fe47-166b-4fa0-9982-afffd73320b8","Type":"ContainerStarted","Data":"e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6"}
Oct 31 01:03:34.727710 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.727682    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vkb7h" event={"ID":"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603","Type":"ContainerStarted","Data":"db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89"}
Oct 31 01:03:34.729495 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.729438    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6h6v" event={"ID":"caa76e03-b876-4bc5-a0ec-aab3cced55f0","Type":"ContainerStarted","Data":"bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c"}
Oct 31 01:03:34.731064 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.731035    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg" event={"ID":"fb29d15c-7076-454a-a335-22e9b8097c4f","Type":"ContainerStarted","Data":"c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997"}
Oct 31 01:03:34.732467 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.732443    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerStarted","Data":"297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20"}
Oct 31 01:03:34.733569 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.733543    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jjdkz" event={"ID":"de7aadea-e0a6-4086-ae7a-a8f7794ad264","Type":"ContainerStarted","Data":"0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35"}
Oct 31 01:03:34.734921 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.734882    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68tmm" event={"ID":"e9562f13-b930-4947-9aac-9379535e8fe6","Type":"ContainerStarted","Data":"73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7"}
Oct 31 01:03:34.819439 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.819403    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:33 +0000 UTC" deadline="2028-06-04 12:12:32.243095017 +0000 UTC"
Oct 31 01:03:34.819439 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:34.819432    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13979h8m57.423665672s"
Oct 31 01:03:35.314905 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.314865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:35.315089 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.315068    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:35.315156 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.315137    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:37.315117952 +0000 UTC m=+6.099250886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:35.415597 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.415548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:35.415784 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.415705    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:35.415784 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.415771    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:03:37.41575414 +0000 UTC m=+6.199887063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:35.516620 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.516530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:35.516820 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.516780    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:35.516884 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.516829    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:35.516884 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.516855    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:35.516986 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.516919    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:37.516900624 +0000 UTC m=+6.301033556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:35.714912 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.714804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:35.715093 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.714958    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:35.715093 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.715036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:35.715244 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.715133    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:35.715244 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.715184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:35.715344 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:35.715255    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:35.820007 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.819892    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:33 +0000 UTC" deadline="2028-07-29 18:12:50.158848204 +0000 UTC"
Oct 31 01:03:35.820007 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:35.819958    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15305h9m14.338894531s"
Oct 31 01:03:37.332034 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.331671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:37.332499 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.332093    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:37.332499 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.332160    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.332139762 +0000 UTC m=+10.116272696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:37.432864 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.432827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:37.433003 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.432984    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:37.433093 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.433047    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.433031049 +0000 UTC m=+10.217163988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:37.534554 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.533926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:37.534554 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.534089    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:37.534554 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.534144    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:37.534554 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.534157    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:37.534554 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.534210    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.534192823 +0000 UTC m=+10.318325750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:37.716593 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.716565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:37.716800 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.716741    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:37.716929 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.716808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:37.716929 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.716878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:37.716929 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.716918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:37.717079 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:37.716975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:37.758780 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.758741    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"fafe7c61d6af085c60e65c5248ab0754","Type":"ContainerStarted","Data":"be0a86f8f7f2590fc573bd7cb3ecff27f171b442ce5498951a448cb61955a74c"}
Oct 31 01:03:37.763570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.763519    2577 generic.go:356] "Generic (PLEG): container finished" podID="8ed86943c8c1fe4dccb7ac2c7a181775" containerID="e621b79a12a4ee8ca0b6947930ae18faed019554e8c949ee7df3a436cfd6291b" exitCode=0
Oct 31 01:03:37.763761 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.763623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"8ed86943c8c1fe4dccb7ac2c7a181775","Type":"ContainerDied","Data":"e621b79a12a4ee8ca0b6947930ae18faed019554e8c949ee7df3a436cfd6291b"}
Oct 31 01:03:37.768776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.767829    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerStarted","Data":"b89359fb9997023ab2cae6f9871d845164f36d61bb782f660d851a580ee2484e"}
Oct 31 01:03:37.985447 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:37.985386    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal" podStartSLOduration=4.985365866 podStartE2EDuration="4.985365866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:37.871488207 +0000 UTC m=+6.655621147" watchObservedRunningTime="2026-10-31 01:03:37.985365866 +0000 UTC m=+6.769498802"
Oct 31 01:03:38.776026 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:38.775986    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"8ed86943c8c1fe4dccb7ac2c7a181775","Type":"ContainerStarted","Data":"f5f3c108c75a0d2e9490fabda37d5d0611692bf39c79a522f8b57b44b280556e"}
Oct 31 01:03:38.780179 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:38.780051    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="b89359fb9997023ab2cae6f9871d845164f36d61bb782f660d851a580ee2484e" exitCode=0
Oct 31 01:03:38.780734 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:38.780709    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerDied","Data":"b89359fb9997023ab2cae6f9871d845164f36d61bb782f660d851a580ee2484e"}
Oct 31 01:03:39.010991 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:39.010701    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" podStartSLOduration=6.010679414 podStartE2EDuration="6.010679414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:38.891168925 +0000 UTC m=+7.675301865" watchObservedRunningTime="2026-10-31 01:03:39.010679414 +0000 UTC m=+7.794812354"
Oct 31 01:03:39.715814 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:39.715783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:39.715982 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:39.715914    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:39.716053 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:39.716014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:39.716127 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:39.716107    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:39.716271 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:39.716255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:39.716379 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:39.716361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:39.788564 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:39.788524    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6h6v" event={"ID":"caa76e03-b876-4bc5-a0ec-aab3cced55f0","Type":"ContainerStarted","Data":"e9bf6b25f68fd1910ead8c1581b231c123a7bbb1405b6ef007086af58c915c77"}
Oct 31 01:03:39.904170 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:39.904107    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j6h6v" podStartSLOduration=3.147558102 podStartE2EDuration="7.904085925s" totalImagesPullingTime="4.756527823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:39.903855199 +0000 UTC m=+8.687988159" watchObservedRunningTime="2026-10-31 01:03:39.904085925 +0000 UTC m=+8.688218866"
Oct 31 01:03:40.793338 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:40.793306    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vkb7h" event={"ID":"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603","Type":"ContainerStarted","Data":"c92bcf7d9985c44dcdb23f9a7ab70bcaea96b56bb2bbbcd70ca3a10048d5a90d"}
Oct 31 01:03:40.796512 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:40.796483    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg" event={"ID":"fb29d15c-7076-454a-a335-22e9b8097c4f","Type":"ContainerStarted","Data":"f0863562163cc4adce956832f28ae3cceeea07470d965fe0017aff2cc9ff7276"}
Oct 31 01:03:40.912419 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:40.910394    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vkb7h" podStartSLOduration=3.228713798 podStartE2EDuration="8.910363897s" totalImagesPullingTime="5.681650099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:40.909767364 +0000 UTC m=+9.693900331" watchObservedRunningTime="2026-10-31 01:03:40.910363897 +0000 UTC m=+9.694496862"
Oct 31 01:03:41.371618 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:41.370723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:41.371618 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.370997    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:41.371618 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.371142    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:49.371120678 +0000 UTC m=+18.155253611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:41.472691 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:41.472048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:41.472691 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.472219    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:41.472691 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.472289    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:03:49.472268494 +0000 UTC m=+18.256401431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:41.574211 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:41.573329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:41.574211 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.573587    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:41.574211 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.573609    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:41.574211 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.573623    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:41.574211 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.573700    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:49.573682539 +0000 UTC m=+18.357815471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:41.714560 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:41.713896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:41.714560 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.714050    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:41.714560 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:41.714449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:41.714560 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.714551    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:41.715039 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:41.714612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:41.715039 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:41.714697    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:42.332506 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:42.332473    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:03:42.677201 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:42.677066    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:03:42.332493838Z","UUID":"273a9cce-2706-4a41-b5c5-b35938d2e7d6","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:03:42.678666 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:42.678626    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:03:42.678666 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:42.678668    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:03:42.807943 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:42.807832    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg" event={"ID":"fb29d15c-7076-454a-a335-22e9b8097c4f","Type":"ContainerStarted","Data":"4e38620dc761f06eed934f84439476deac4c0fbcebe9e43c56050b09175d0e78"}
Oct 31 01:03:42.810116 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:42.810088    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68tmm" event={"ID":"e9562f13-b930-4947-9aac-9379535e8fe6","Type":"ContainerStarted","Data":"50a0cf5e653a6f167faeeef0239339a930ab1eb6b852525b89208158df7d81f6"}
Oct 31 01:03:43.189619 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.189579    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:43.190252 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.190211    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:43.303137 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.303038    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-68tmm" podStartSLOduration=2.93748197 podStartE2EDuration="11.3030252s" totalImagesPullingTime="8.36554323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:42.933866864 +0000 UTC m=+11.717999841" watchObservedRunningTime="2026-10-31 01:03:43.3030252 +0000 UTC m=+12.087158138"
Oct 31 01:03:43.716901 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.716868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:43.717474 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:43.716996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:43.717474 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.717372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:43.717588 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:43.717472    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:43.717588 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.717533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:43.717682 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:43.717601    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:43.815094 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.815005    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="59cb839dd41293b00297f334d6fcbf5cc4dda349688f1eaf88de46f2dbbcf95f" exitCode=0
Oct 31 01:03:43.815230 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.815097    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerDied","Data":"59cb839dd41293b00297f334d6fcbf5cc4dda349688f1eaf88de46f2dbbcf95f"}
Oct 31 01:03:43.819037 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:43.819014    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jjdkz" event={"ID":"de7aadea-e0a6-4086-ae7a-a8f7794ad264","Type":"ContainerStarted","Data":"4043ba2add2e7956ceb1f97f1eb01e9ec19a8d2556db91c34afb15c8ba3346e2"}
Oct 31 01:03:44.822722 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:44.822441    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrnxg" event={"ID":"2a09fe47-166b-4fa0-9982-afffd73320b8","Type":"ContainerStarted","Data":"6332c16598581b9a6db124988f9bdd88ef99e02c9aa593e85ba50d339c23958d"}
Oct 31 01:03:44.825017 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:44.824979    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg" event={"ID":"fb29d15c-7076-454a-a335-22e9b8097c4f","Type":"ContainerStarted","Data":"56cf29c4779fb9b99b14859238d7b37d24b8fbbc315550151bc1f98aba80e30e"}
Oct 31 01:03:44.937276 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:44.937235    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jjdkz" podStartSLOduration=4.161812316 podStartE2EDuration="12.937220457s" totalImagesPullingTime="8.775408141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:44.05482885 +0000 UTC m=+12.838961789" watchObservedRunningTime="2026-10-31 01:03:44.937220457 +0000 UTC m=+13.721353396"
Oct 31 01:03:44.937407 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:44.937357    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rrnxg" podStartSLOduration=4.431449737 podStartE2EDuration="12.937350407s" totalImagesPullingTime="8.50590067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:44.936934495 +0000 UTC m=+13.721067435" watchObservedRunningTime="2026-10-31 01:03:44.937350407 +0000 UTC m=+13.721483347"
Oct 31 01:03:45.714388 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:45.714332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:45.714621 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:45.714345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:45.714621 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:45.714587    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:45.714786 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:45.714634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:45.714838 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:45.714774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:45.714895 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:45.714864    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:45.831033 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:45.830985    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="8309b6b79befd33427b8496efa62679666407b972566891d3077d4352c3fd3bc" exitCode=0
Oct 31 01:03:45.832006 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:45.831076    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerDied","Data":"8309b6b79befd33427b8496efa62679666407b972566891d3077d4352c3fd3bc"}
Oct 31 01:03:45.833447 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:45.833404    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-psw7z" event={"ID":"7c3cee94-19dc-4e56-b7a2-256765111acb","Type":"ContainerStarted","Data":"de9e01ec4090737b483c15be6d26dc9d978745e8b80eae5a08fdbfe47dfc875c"}
Oct 31 01:03:46.001580 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:46.001527    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg" podStartSLOduration=4.555975381 podStartE2EDuration="14.001506678s" totalImagesPullingTime="9.445531297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:45.056952197 +0000 UTC m=+13.841085129" watchObservedRunningTime="2026-10-31 01:03:46.001506678 +0000 UTC m=+14.785639620"
Oct 31 01:03:46.118496 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:46.118434    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-psw7z" podStartSLOduration=3.188618335 podStartE2EDuration="14.118418483s" totalImagesPullingTime="10.929800148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:46.117826484 +0000 UTC m=+14.901959423" watchObservedRunningTime="2026-10-31 01:03:46.118418483 +0000 UTC m=+14.902551422"
Oct 31 01:03:47.713983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:47.713945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:47.714708 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:47.713951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:47.714708 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:47.714075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:47.714708 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:47.714081    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:47.714708 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:47.714175    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:47.714708 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:47.714250    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:47.841175 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:47.841137    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="38c828ed6ea35a73bad82cbd882e226d576f366a314271c7c56b23051b95b88e" exitCode=0
Oct 31 01:03:47.841352 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:47.841178    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerDied","Data":"38c828ed6ea35a73bad82cbd882e226d576f366a314271c7c56b23051b95b88e"}
Oct 31 01:03:49.304646 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.304358    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:49.305085 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.304812    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:49.305318 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.305277    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:03:49.438150 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.438106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:49.438299 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.438262    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:49.438363 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.438333    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:05.438312176 +0000 UTC m=+34.222445109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:49.538751 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.538717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:49.538889 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.538835    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:49.538921 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.538894    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:04:05.538880014 +0000 UTC m=+34.323012931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:49.640248 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.639975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:49.640248 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.640127    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:49.640248 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.640157    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:49.640248 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.640172    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:49.640248 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.640230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:05.640211962 +0000 UTC m=+34.424344893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:49.713401 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.713374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:49.713401 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.713390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:49.713552 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.713535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:49.713597 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.713545    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:49.713720 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.713677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:49.713782 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:49.713729    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:49.848795 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.848770    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"df08de7643feeb4e0aff93671bcc21cf155291a67b59600deac041ad01348c7a"}
Oct 31 01:03:49.848886 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.848838    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"ff8eca4477ade824a849bfb3e69690bc7f34a5394c5f40e449b0c7fc71ad5196"}
Oct 31 01:03:49.848886 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.848870    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"83a708d64f3a7a0a7ba0536471f95527af06ab6d38bdecb6e379efc9905c1a3a"}
Oct 31 01:03:49.848974 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:49.848896    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"9e85948f7c1b39e13fc9c0a16de1527541233afd78f55586e60fa022ae4dff7d"}
Oct 31 01:03:50.854304 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:50.854085    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"a9677798ffe2734c74ab1e859a0380900dc14e41a0a8159820612b4d59220c8c"}
Oct 31 01:03:50.854851 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:50.854397    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"e0e8e31af6c1a622aff92d027519b80ba1190db950eb88ae740b5b9496247264"}
Oct 31 01:03:50.856730 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:50.856704    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="f3f513c01cb242c79ba14e31e9161d24c2c09dbad86e38369f929d6fdf740b83" exitCode=0
Oct 31 01:03:50.856844 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:50.856743    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerDied","Data":"f3f513c01cb242c79ba14e31e9161d24c2c09dbad86e38369f929d6fdf740b83"}
Oct 31 01:03:51.714870 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:51.714837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:51.715013 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:51.714948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:51.715013 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:51.714965    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:51.715013 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:51.715002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:51.715120 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:51.715065    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:51.715165 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:51.715130    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:51.861828 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:51.861797    2577 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="a0b4ce9a252479a912627f8c64fb4e4ee27db4cb0e2b24e60adbd6c5c226885a" exitCode=0
Oct 31 01:03:51.862390 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:51.861854    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerDied","Data":"a0b4ce9a252479a912627f8c64fb4e4ee27db4cb0e2b24e60adbd6c5c226885a"}
Oct 31 01:03:52.789522 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:52.789496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jjdkz_de7aadea-e0a6-4086-ae7a-a8f7794ad264/dns-node-resolver/0.log"
Oct 31 01:03:52.867834 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:52.867797    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" event={"ID":"1e8405b7-4f76-495c-b9fc-04affdf8d1f8","Type":"ContainerStarted","Data":"8b45e05b8dcc667918477e1b24ae47507110c7b9e46c5cc7031013fed063b7ae"}
Oct 31 01:03:52.870879 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:52.870854    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"f59bcbd5fad04dffbf82903d5846300a3812765a341a788ee73a4e96b1f4ed2f"}
Oct 31 01:03:53.713719 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:53.713681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:53.713719 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:53.713722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:53.713928 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:53.713684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:53.713928 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:53.713802    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:53.713985 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:53.713920    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:53.714049 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:53.714029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:53.806982 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:53.806956    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vkb7h_c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603/node-ca/0.log"
Oct 31 01:03:54.880996 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:54.880767    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" event={"ID":"2c464f6d-9424-478f-8b7c-c64b564d816f","Type":"ContainerStarted","Data":"731ab52a961e8df0ee7a3d7804b85eee78824b09a456f2400bbff61c3a90abce"}
Oct 31 01:03:54.881606 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:54.881088    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:54.897911 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:54.897885    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:55.155854 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.155733    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9lpwg" podStartSLOduration=10.280174433 podStartE2EDuration="23.155718358s" totalImagesPullingTime="12.875543925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:53.000304911 +0000 UTC m=+21.784437867" watchObservedRunningTime="2026-10-31 01:03:55.155718358 +0000 UTC m=+23.939851304"
Oct 31 01:03:55.156020 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.156000    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" podStartSLOduration=7.637178368 podStartE2EDuration="23.155993512s" totalImagesPullingTime="15.518815144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:55.105841864 +0000 UTC m=+23.889974802" watchObservedRunningTime="2026-10-31 01:03:55.155993512 +0000 UTC m=+23.940126495"
Oct 31 01:03:55.714443 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.714409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:55.714443 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.714434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:55.714672 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.714466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:55.714672 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:55.714551    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:55.714672 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:55.714620    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:55.714796 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:55.714716    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:55.882863 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.882837    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:55.883281 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.883262    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:55.898752 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:55.898725    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:56.884759 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:56.884730    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:57.714057 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:57.713993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:57.714302 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:57.713993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:57.714302 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:57.714183    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:57.714302 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:57.713993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:57.714302 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:57.714216    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:57.714302 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:57.714285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:57.887528 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:57.887484    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:59.713842 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.713797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:03:59.714430 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.713810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:03:59.714430 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:59.713931    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:03:59.714430 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.713810    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:03:59.714430 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:59.713972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:03:59.714430 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:03:59.714055    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:03:59.779247 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.779218    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:03:59.779441 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.779428    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:59.789035 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.789001    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" podUID="2c464f6d-9424-478f-8b7c-c64b564d816f" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 31 01:03:59.798272 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:03:59.798243    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp" podUID="2c464f6d-9424-478f-8b7c-c64b564d816f" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 31 01:04:01.716332 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:01.716297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:01.716332 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:01.716313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:01.716332 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:01.716336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:01.716824 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:01.716430    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:01.716824 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:01.716538    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:01.716824 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:01.716619    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:03.714324 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:03.714295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:03.714324 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:03.714328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:03.714803 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:03.714294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:03.714803 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:03.714408    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:03.714803 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:03.714470    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:03.714803 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:03.714556    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:05.453884 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:05.453843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:05.454306 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.453990    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:04:05.454306 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.454048    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs podName:bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:37.454035242 +0000 UTC m=+66.238168173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs") pod "network-metrics-daemon-rdsvq" (UID: "bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:04:05.554726 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:05.554692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:05.554891 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.554825    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:04:05.554955 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.554892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret podName:2e497701-d9b9-46b7-a5db-db294421d12f nodeName:}" failed. No retries permitted until 2026-10-31 01:04:37.554873236 +0000 UTC m=+66.339006169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret") pod "global-pull-secret-syncer-pf5kd" (UID: "2e497701-d9b9-46b7-a5db-db294421d12f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:04:05.654972 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:05.654940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:05.655131 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.655060    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:04:05.655131 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.655082    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:04:05.655131 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.655095    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jmslr for pod openshift-network-diagnostics/network-check-target-fczdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:04:05.655227 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.655143    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr podName:792f66b0-e2d9-47c0-92bc-071ba29d52e5 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:37.655129775 +0000 UTC m=+66.439262692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmslr" (UniqueName: "kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr") pod "network-check-target-fczdh" (UID: "792f66b0-e2d9-47c0-92bc-071ba29d52e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:04:05.714112 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:05.714052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:05.714202 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:05.714052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:05.714202 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.714179    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:05.714295 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:05.714061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:05.714411 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.714270    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:05.714411 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:05.714362    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:07.713938 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:07.713896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:07.714340 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:07.713900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:07.714340 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:07.714018    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:07.714340 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:07.713904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:07.714340 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:07.714079    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:07.714340 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:07.714170    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:09.713770 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:09.713729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:09.713770 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:09.713762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:09.714276 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:09.713729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:09.714276 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:09.713851    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:09.714276 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:09.713950    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:09.714276 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:09.714014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:11.715710 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:11.715681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:11.716138 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:11.715721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:11.716138 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:11.715804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:11.716138 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:11.715810    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:11.716138 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:11.715881    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:11.716138 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:11.715927    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:13.714350 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.714262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:13.714822 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.714275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:13.714822 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:13.714403    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:13.714822 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.714438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:13.714822 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:13.714563    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:13.714822 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:13.714641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:13.894776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.894745    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pf5kd"]
Oct 31 01:04:13.898707 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.898680    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fczdh"]
Oct 31 01:04:13.898809 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.898729    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdsvq"]
Oct 31 01:04:13.921856 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.921830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:13.921979 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.921831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:13.921979 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:13.921960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:13.922061 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:13.921830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:13.922061 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:13.922017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:13.922138 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:13.922082    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:15.713859 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:15.713624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:15.714394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:15.713624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:15.714394 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:15.713940    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:15.714394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:15.713727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:15.714394 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:15.713996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:15.714394 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:15.714093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:17.713776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:17.713737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:17.713776 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:17.713758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:17.714318 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:17.713865    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pf5kd" podUID="2e497701-d9b9-46b7-a5db-db294421d12f"
Oct 31 01:04:17.714318 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:17.713957    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fczdh" podUID="792f66b0-e2d9-47c0-92bc-071ba29d52e5"
Oct 31 01:04:17.714318 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:17.713999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:17.714318 ip-10-0-141-86 kubenswrapper[2577]: E1031 01:04:17.714066    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdsvq" podUID="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81"
Oct 31 01:04:18.464039 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.463968    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeReady"
Oct 31 01:04:18.464187 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.464099    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 01:04:18.588006 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.587970    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ntdx6"]
Oct 31 01:04:18.589622 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.589606    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.638469 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.638441    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mr6h9"]
Oct 31 01:04:18.642221 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.642189    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hmdlr"]
Oct 31 01:04:18.642450 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.642358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.642450 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.642415    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:04:18.642606 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.642453    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:04:18.642606 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.642507    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:04:18.642606 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.642590    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xgzbp\""
Oct 31 01:04:18.643955 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.643937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.655705 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.655685    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ntdx6"]
Oct 31 01:04:18.658755 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.658732    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mr6h9"]
Oct 31 01:04:18.688492 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.688466    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmdlr"]
Oct 31 01:04:18.734963 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.734879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtpv\" (UniqueName: \"kubernetes.io/projected/f051a847-0102-43c1-8fd4-7885e50459c2-kube-api-access-kbtpv\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.734963 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.734916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1608abd9-fd52-44e3-a945-eccf76f178aa-metrics-tls\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.734963 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.734932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1608abd9-fd52-44e3-a945-eccf76f178aa-tmp-dir\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-data-volume\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f051a847-0102-43c1-8fd4-7885e50459c2-cert\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29d76\" (UniqueName: \"kubernetes.io/projected/1608abd9-fd52-44e3-a945-eccf76f178aa-kube-api-access-29d76\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-crio-socket\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1608abd9-fd52-44e3-a945-eccf76f178aa-config-volume\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xcq\" (UniqueName: \"kubernetes.io/projected/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-api-access-q9xcq\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.735457 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.735349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.742825 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.742803    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:04:18.742825 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.742813    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:04:18.743003 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.742983    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:04:18.743142 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.743129    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:04:18.743197 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.743161    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sjdn\""
Oct 31 01:04:18.743197 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.743154    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:04:18.743197 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.743186    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:04:18.743299 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.743267    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvb7t\""
Oct 31 01:04:18.835793 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.835758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-data-volume\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.835793 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.835791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.835998 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.835811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f051a847-0102-43c1-8fd4-7885e50459c2-cert\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.835998 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.835952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29d76\" (UniqueName: \"kubernetes.io/projected/1608abd9-fd52-44e3-a945-eccf76f178aa-kube-api-access-29d76\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.835998 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.835991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-crio-socket\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.836133 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1608abd9-fd52-44e3-a945-eccf76f178aa-config-volume\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.836133 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xcq\" (UniqueName: \"kubernetes.io/projected/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-api-access-q9xcq\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.836133 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.836133 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtpv\" (UniqueName: \"kubernetes.io/projected/f051a847-0102-43c1-8fd4-7885e50459c2-kube-api-access-kbtpv\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.836317 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836149    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-data-volume\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.836317 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836227    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-crio-socket\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.836317 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1608abd9-fd52-44e3-a945-eccf76f178aa-metrics-tls\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.836317 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1608abd9-fd52-44e3-a945-eccf76f178aa-tmp-dir\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.836594 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.836689 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1608abd9-fd52-44e3-a945-eccf76f178aa-tmp-dir\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.836938 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.836915    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1608abd9-fd52-44e3-a945-eccf76f178aa-config-volume\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.840000 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.839977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1608abd9-fd52-44e3-a945-eccf76f178aa-metrics-tls\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.840080 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.840002    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.840080 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.840012    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f051a847-0102-43c1-8fd4-7885e50459c2-cert\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.894338 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.894298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29d76\" (UniqueName: \"kubernetes.io/projected/1608abd9-fd52-44e3-a945-eccf76f178aa-kube-api-access-29d76\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:18.894338 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.894328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtpv\" (UniqueName: \"kubernetes.io/projected/f051a847-0102-43c1-8fd4-7885e50459c2-kube-api-access-kbtpv\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.894507 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.894484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xcq\" (UniqueName: \"kubernetes.io/projected/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-api-access-q9xcq\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.900394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.900371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:18.954031 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.954001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:18.959907 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:18.958116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:19.291328 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.291248    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ntdx6"]
Oct 31 01:04:19.325799 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:19.325773    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf051a847_0102_43c1_8fd4_7885e50459c2.slice/crio-f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88 WatchSource:0}: Error finding container f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88: Status 404 returned error can't find the container with id f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88
Oct 31 01:04:19.337989 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.337968    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mr6h9"]
Oct 31 01:04:19.340046 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:19.340026    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5adc1e9b_39f1_4a9e_bbc6_1a5f8fdf6373.slice/crio-ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e WatchSource:0}: Error finding container ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e: Status 404 returned error can't find the container with id ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e
Oct 31 01:04:19.388098 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.388075    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmdlr"]
Oct 31 01:04:19.390716 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:19.390690    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1608abd9_fd52_44e3_a945_eccf76f178aa.slice/crio-c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d WatchSource:0}: Error finding container c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d: Status 404 returned error can't find the container with id c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d
Oct 31 01:04:19.716666 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.716566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:19.716666 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.716609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:19.716873 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.716697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:19.770402 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770376    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:04:19.770815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770499    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:04:19.770815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770577    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:04:19.770815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770593    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:04:19.770815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770614    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:04:19.770815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770680    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:04:19.770815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.770693    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:04:19.937465 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.937433    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mr6h9" event={"ID":"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373","Type":"ContainerStarted","Data":"ef4b01600a35c2646ffca49c8c2996b6d19052dd5b07b5ecded767a80d5bb438"}
Oct 31 01:04:19.937643 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.937496    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mr6h9" event={"ID":"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373","Type":"ContainerStarted","Data":"ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e"}
Oct 31 01:04:19.938583 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.938552    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ntdx6" event={"ID":"f051a847-0102-43c1-8fd4-7885e50459c2","Type":"ContainerStarted","Data":"f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88"}
Oct 31 01:04:19.939584 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:19.939566    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmdlr" event={"ID":"1608abd9-fd52-44e3-a945-eccf76f178aa","Type":"ContainerStarted","Data":"c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d"}
Oct 31 01:04:20.944708 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:20.944605    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mr6h9" event={"ID":"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373","Type":"ContainerStarted","Data":"bc5c10c4ad720f8aae440126c090a2bb3df56c04beeabdea8e5fbf30d5495383"}
Oct 31 01:04:22.591538 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.591502    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"]
Oct 31 01:04:22.594105 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.594080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.696518 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.696444    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:04:22.696518 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.696447    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 01:04:22.696793 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.696776    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:04:22.696793 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.696787    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:04:22.696888 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.696776    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:04:22.696888 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.696820    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xs8jk\""
Oct 31 01:04:22.700804 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.700785    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"]
Oct 31 01:04:22.701283 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.701265    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"]
Oct 31 01:04:22.703214 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.703196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.730936 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.730910    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tm9rb"]
Oct 31 01:04:22.732556 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.732542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.756483 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.756458    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7hnm9\""
Oct 31 01:04:22.756760 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.756741    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 01:04:22.756824 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.756741    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:04:22.757097 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.757085    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 01:04:22.764070 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.764039    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"]
Oct 31 01:04:22.769557 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.769536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65337868-194c-4e3f-9e3f-7e96c35c6761-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.769697 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.769568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9jn\" (UniqueName: \"kubernetes.io/projected/65337868-194c-4e3f-9e3f-7e96c35c6761-kube-api-access-6r9jn\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.769697 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.769611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.769697 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.769684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.811866 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.811833    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:04:22.856983 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.856955    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:04:22.857149 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.857017    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:04:22.857149 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.857031    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ln2ts\""
Oct 31 01:04:22.870500 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpn68\" (UniqueName: \"kubernetes.io/projected/961e1a63-66d7-4c5f-924e-27d9d56d6862-kube-api-access-xpn68\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.870500 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.870710 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9jn\" (UniqueName: \"kubernetes.io/projected/65337868-194c-4e3f-9e3f-7e96c35c6761-kube-api-access-6r9jn\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.870710 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05380ea7-efbf-4c24-9163-97a4243365df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.870710 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-textfile\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.870710 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.870830 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.870863 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.870899 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-wtmp\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.870933 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-metrics-client-ca\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.871020 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.870997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-accelerators-collector-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.871079 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.871126 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84vl\" (UniqueName: \"kubernetes.io/projected/05380ea7-efbf-4c24-9163-97a4243365df-kube-api-access-x84vl\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.871158 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.871158 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.871229 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-root\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.871229 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-sys\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.871229 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-tls\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.871349 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65337868-194c-4e3f-9e3f-7e96c35c6761-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.871891 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.871852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65337868-194c-4e3f-9e3f-7e96c35c6761-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.873474 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.873450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.873598 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.873585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.927518 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.927482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9jn\" (UniqueName: \"kubernetes.io/projected/65337868-194c-4e3f-9e3f-7e96c35c6761-kube-api-access-6r9jn\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:22.952177 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.952092    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmdlr" event={"ID":"1608abd9-fd52-44e3-a945-eccf76f178aa","Type":"ContainerStarted","Data":"8cba26d45cafc479eebe69a6bec51e594f7d6447f034f61677e286f47984abe5"}
Oct 31 01:04:22.952177 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.952149    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmdlr" event={"ID":"1608abd9-fd52-44e3-a945-eccf76f178aa","Type":"ContainerStarted","Data":"e552de738d9b1c4b651aec41b92799fb3e086ecbf16c261821a5b1f391f2a1a0"}
Oct 31 01:04:22.952371 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.952225    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:22.954125 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.954099    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mr6h9" event={"ID":"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373","Type":"ContainerStarted","Data":"5acb0137425de7aa2b78e709e21d8ca652ac226bb81e1dd15e29a088fff61caf"}
Oct 31 01:04:22.955562 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.955525    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ntdx6" event={"ID":"f051a847-0102-43c1-8fd4-7885e50459c2","Type":"ContainerStarted","Data":"08a30adccaac7e7cf3751e814ae5f81ccc193d2931145a2a255447bd031c2fac"}
Oct 31 01:04:22.971661 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-sys\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.971764 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-tls\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.971764 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpn68\" (UniqueName: \"kubernetes.io/projected/961e1a63-66d7-4c5f-924e-27d9d56d6862-kube-api-access-xpn68\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.971854 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971766    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-sys\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.971854 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.971940 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05380ea7-efbf-4c24-9163-97a4243365df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.971940 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-textfile\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972025 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.971989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.972078 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.972078 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-wtmp\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972172 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-metrics-client-ca\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972172 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-accelerators-collector-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972172 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.972311 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84vl\" (UniqueName: \"kubernetes.io/projected/05380ea7-efbf-4c24-9163-97a4243365df-kube-api-access-x84vl\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.972311 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-textfile\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972311 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972311 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-root\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972311 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05380ea7-efbf-4c24-9163-97a4243365df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.972526 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972325    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-root\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972526 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-wtmp\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972735 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972702    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.972847 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972735    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-accelerators-collector-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.972923 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.972885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-metrics-client-ca\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.973234 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.973187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.974392 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.974367    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-tls\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.974975 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.974950    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:22.975110 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.975094    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:22.975168 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:22.975132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:23.028982 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.028950    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpn68\" (UniqueName: \"kubernetes.io/projected/961e1a63-66d7-4c5f-924e-27d9d56d6862-kube-api-access-xpn68\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:23.029122 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.029102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84vl\" (UniqueName: \"kubernetes.io/projected/05380ea7-efbf-4c24-9163-97a4243365df-kube-api-access-x84vl\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:23.044144 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.044120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:23.053179 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:23.053151    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961e1a63_66d7_4c5f_924e_27d9d56d6862.slice/crio-c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f WatchSource:0}: Error finding container c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f: Status 404 returned error can't find the container with id c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f
Oct 31 01:04:23.092021 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.091970    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ntdx6" podStartSLOduration=1.984585988 podStartE2EDuration="5.091954169s" totalImagesPullingTime="3.107368181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:23.091368314 +0000 UTC m=+51.875501256" watchObservedRunningTime="2026-10-31 01:04:23.091954169 +0000 UTC m=+51.876087101"
Oct 31 01:04:23.206280 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.206189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:23.210045 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.209997    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mr6h9" podStartSLOduration=2.088900043 podStartE2EDuration="5.20998267s" totalImagesPullingTime="3.121082627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:04:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:23.209602769 +0000 UTC m=+51.993735708" watchObservedRunningTime="2026-10-31 01:04:23.20998267 +0000 UTC m=+51.994115658"
Oct 31 01:04:23.314760 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.314724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:23.324824 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.324764    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hmdlr" podStartSLOduration=2.625197989 podStartE2EDuration="5.324747137s" totalImagesPullingTime="2.699549148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:23.323157684 +0000 UTC m=+52.107290624" watchObservedRunningTime="2026-10-31 01:04:23.324747137 +0000 UTC m=+52.108880079"
Oct 31 01:04:23.623224 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.623156    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"]
Oct 31 01:04:23.625487 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:23.625455    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65337868_194c_4e3f_9e3f_7e96c35c6761.slice/crio-de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3 WatchSource:0}: Error finding container de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3: Status 404 returned error can't find the container with id de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3
Oct 31 01:04:23.692052 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.692007    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"]
Oct 31 01:04:23.708961 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:23.708934    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05380ea7_efbf_4c24_9163_97a4243365df.slice/crio-ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261 WatchSource:0}: Error finding container ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261: Status 404 returned error can't find the container with id ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261
Oct 31 01:04:23.962962 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.962869    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5" event={"ID":"05380ea7-efbf-4c24-9163-97a4243365df","Type":"ContainerStarted","Data":"ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261"}
Oct 31 01:04:23.964088 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.964046    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tm9rb" event={"ID":"961e1a63-66d7-4c5f-924e-27d9d56d6862","Type":"ContainerStarted","Data":"c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f"}
Oct 31 01:04:23.965969 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.965939    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp" event={"ID":"65337868-194c-4e3f-9e3f-7e96c35c6761","Type":"ContainerStarted","Data":"a34fc5ae2e6540c1de70f1be0f8852c73235e09ae34e5b96e9083c99aeaf931b"}
Oct 31 01:04:23.966144 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.965992    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp" event={"ID":"65337868-194c-4e3f-9e3f-7e96c35c6761","Type":"ContainerStarted","Data":"3bd7922627fd3c4dff1231ddba876c1f76f2b10491905450da16aaf2d830bcf2"}
Oct 31 01:04:23.966144 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:23.966015    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp" event={"ID":"65337868-194c-4e3f-9e3f-7e96c35c6761","Type":"ContainerStarted","Data":"de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3"}
Oct 31 01:04:24.972614 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:24.972565    2577 generic.go:356] "Generic (PLEG): container finished" podID="961e1a63-66d7-4c5f-924e-27d9d56d6862" containerID="1e0f173caccdc0c3e21a41b406c6baa72384c678b9e05f5e81036b9c2330d938" exitCode=0
Oct 31 01:04:24.973066 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:24.972623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tm9rb" event={"ID":"961e1a63-66d7-4c5f-924e-27d9d56d6862","Type":"ContainerDied","Data":"1e0f173caccdc0c3e21a41b406c6baa72384c678b9e05f5e81036b9c2330d938"}
Oct 31 01:04:25.352165 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.350046    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"]
Oct 31 01:04:25.354877 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.353356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.408499 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408446    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 01:04:25.408499 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408469    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mdbs5\""
Oct 31 01:04:25.408499 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408486    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 01:04:25.408499 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408447    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 01:04:25.408830 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408518    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 01:04:25.408830 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408583    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8c8qv8uqffkk1\""
Oct 31 01:04:25.408830 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.408611    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 01:04:25.419641 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.419614    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"]
Oct 31 01:04:25.493266 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493429 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911363c3-235f-4667-b12c-94089715f1bc-metrics-client-ca\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493429 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493544 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493544 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493643 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmx7m\" (UniqueName: \"kubernetes.io/projected/911363c3-235f-4667-b12c-94089715f1bc-kube-api-access-mmx7m\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493724 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911363c3-235f-4667-b12c-94089715f1bc-tmp\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493808 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.493953 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.493824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-grpc-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594350 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-grpc-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911363c3-235f-4667-b12c-94089715f1bc-metrics-client-ca\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmx7m\" (UniqueName: \"kubernetes.io/projected/911363c3-235f-4667-b12c-94089715f1bc-kube-api-access-mmx7m\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594836 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911363c3-235f-4667-b12c-94089715f1bc-tmp\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.594836 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.594596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.595181 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.595120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911363c3-235f-4667-b12c-94089715f1bc-tmp\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.595411 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.595366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911363c3-235f-4667-b12c-94089715f1bc-metrics-client-ca\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.597141 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.597115    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-grpc-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.597722 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.597698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.597830 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.597718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.597830 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.597767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.597984 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.597966    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.598195 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.598178    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.654761 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.654733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmx7m\" (UniqueName: \"kubernetes.io/projected/911363c3-235f-4667-b12c-94089715f1bc-kube-api-access-mmx7m\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.674613 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.674588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:25.980939 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.980835    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5" event={"ID":"05380ea7-efbf-4c24-9163-97a4243365df","Type":"ContainerStarted","Data":"da595c50d9d736cd000025baeba6e55a56a894351a8a6c5bc12a427bf5fc9b6e"}
Oct 31 01:04:25.980939 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.980897    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5" event={"ID":"05380ea7-efbf-4c24-9163-97a4243365df","Type":"ContainerStarted","Data":"6488aa5159b2ae31e2605fe5de363e2bf56987be611996da7ef06d2daaf0e2ad"}
Oct 31 01:04:25.980939 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.980927    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5" event={"ID":"05380ea7-efbf-4c24-9163-97a4243365df","Type":"ContainerStarted","Data":"f8f7a1ba5d746f2f07ac62f58c2b9e803e87703302513c5815cc870454d6ac3e"}
Oct 31 01:04:25.982981 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.982950    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tm9rb" event={"ID":"961e1a63-66d7-4c5f-924e-27d9d56d6862","Type":"ContainerStarted","Data":"9bf36b71423469964716293f3ab2569749a84d374b7a5e08965c810b38bfe2e1"}
Oct 31 01:04:25.983093 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.983007    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tm9rb" event={"ID":"961e1a63-66d7-4c5f-924e-27d9d56d6862","Type":"ContainerStarted","Data":"de65fe27766540ff1cccf17da5b676d0b7624b01a257cee9e863c1c1e221faad"}
Oct 31 01:04:25.985122 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:25.985099    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp" event={"ID":"65337868-194c-4e3f-9e3f-7e96c35c6761","Type":"ContainerStarted","Data":"8b769f708df9c5f753f6482e057a350499755c364b8f9b32d19392f99b10ca7c"}
Oct 31 01:04:26.100113 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:26.100066    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5" podStartSLOduration=2.209532918 podStartE2EDuration="4.100052577s" totalImagesPullingTime="1.890519659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:26.098896146 +0000 UTC m=+54.883029085" watchObservedRunningTime="2026-10-31 01:04:26.100052577 +0000 UTC m=+54.884185517"
Oct 31 01:04:26.104671 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:26.104627    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"]
Oct 31 01:04:26.107298 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:26.107270    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911363c3_235f_4667_b12c_94089715f1bc.slice/crio-c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440 WatchSource:0}: Error finding container c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440: Status 404 returned error can't find the container with id c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440
Oct 31 01:04:26.219593 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:26.219526    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp" podStartSLOduration=2.661304193 podStartE2EDuration="4.219487168s" totalImagesPullingTime="1.558182975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:26.21876863 +0000 UTC m=+55.002901570" watchObservedRunningTime="2026-10-31 01:04:26.219487168 +0000 UTC m=+55.003620108"
Oct 31 01:04:26.336668 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:26.336590    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tm9rb" podStartSLOduration=2.897639876 podStartE2EDuration="4.33656234s" totalImagesPullingTime="1.366363289s" totalInitContainerRuntime="72.559175ms" isStatefulPod=true podCreationTimestamp="2026-10-31 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:26.335832355 +0000 UTC m=+55.119965292" watchObservedRunningTime="2026-10-31 01:04:26.33656234 +0000 UTC m=+55.120695287"
Oct 31 01:04:26.989036 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:26.989000    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440"}
Oct 31 01:04:27.389901 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.389861    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"]
Oct 31 01:04:27.393009 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.392986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.445084 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.445051    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 01:04:27.445281 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.445106    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wdn2p\""
Oct 31 01:04:27.450217 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.450187    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"]
Oct 31 01:04:27.511030 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.510991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf97\" (UniqueName: \"kubernetes.io/projected/120f0f3f-da9b-4395-807b-a291a37b5e2d-kube-api-access-bkf97\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.511223 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.511200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/120f0f3f-da9b-4395-807b-a291a37b5e2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.611725 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.611674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf97\" (UniqueName: \"kubernetes.io/projected/120f0f3f-da9b-4395-807b-a291a37b5e2d-kube-api-access-bkf97\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.611905 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.611891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/120f0f3f-da9b-4395-807b-a291a37b5e2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.615273 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.615248    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/120f0f3f-da9b-4395-807b-a291a37b5e2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.669306 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.669237    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf97\" (UniqueName: \"kubernetes.io/projected/120f0f3f-da9b-4395-807b-a291a37b5e2d-kube-api-access-bkf97\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.706040 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.706013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:27.759141 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.758892    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"]
Oct 31 01:04:27.763794 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.763770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.818394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.818343    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 01:04:27.818394 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.818382    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 01:04:27.820145 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.820116    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 01:04:27.820468 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.820446    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 01:04:27.820903 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.820870    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vk6v8\""
Oct 31 01:04:27.824815 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.824691    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"]
Oct 31 01:04:27.867515 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.867480    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-36k3cvsjqf5vk\""
Oct 31 01:04:27.914005 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.913967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-tls\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.914191 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.914050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-metrics-server-audit-profiles\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.914191 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.914129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-client-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.914191 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.914167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-client-certs\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.914351 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.914204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.914351 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.914289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8n7\" (UniqueName: \"kubernetes.io/projected/432f223d-4785-4a5c-af6a-921a65b1a08e-kube-api-access-gz8n7\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:27.914351 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:27.914340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/432f223d-4785-4a5c-af6a-921a65b1a08e-audit-log\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015075 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-client-certs\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8n7\" (UniqueName: \"kubernetes.io/projected/432f223d-4785-4a5c-af6a-921a65b1a08e-kube-api-access-gz8n7\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/432f223d-4785-4a5c-af6a-921a65b1a08e-audit-log\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-tls\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-metrics-server-audit-profiles\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015570 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-client-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015950 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/432f223d-4785-4a5c-af6a-921a65b1a08e-audit-log\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.015950 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.015931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.016222 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.016197    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-metrics-server-audit-profiles\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.017812 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.017794    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-client-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.018410 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.018388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-tls\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.018487 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.018387    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-client-certs\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.072480 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.072448    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8n7\" (UniqueName: \"kubernetes.io/projected/432f223d-4785-4a5c-af6a-921a65b1a08e-kube-api-access-gz8n7\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.073604 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.073577    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"]
Oct 31 01:04:28.076122 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:28.076092    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120f0f3f_da9b_4395_807b_a291a37b5e2d.slice/crio-d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579 WatchSource:0}: Error finding container d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579: Status 404 returned error can't find the container with id d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579
Oct 31 01:04:28.078514 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.078484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:28.503778 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.503666    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"]
Oct 31 01:04:28.506930 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:28.506898    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432f223d_4785_4a5c_af6a_921a65b1a08e.slice/crio-db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2 WatchSource:0}: Error finding container db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2: Status 404 returned error can't find the container with id db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2
Oct 31 01:04:28.586980 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.586950    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5d58f5bd-vchll"]
Oct 31 01:04:28.591242 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.591215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.643867 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643836    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ch46h\""
Oct 31 01:04:28.643867 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643860    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 01:04:28.644108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643884    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 01:04:28.644108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643869    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 01:04:28.644108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643906    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 01:04:28.644108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643836    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 01:04:28.644108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643935    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 01:04:28.644108 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.643937    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 01:04:28.651141 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.651113    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5d58f5bd-vchll"]
Oct 31 01:04:28.720552 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.720766 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.720766 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.720864 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.720864 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.720864 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.720989 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.720965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822211 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822211 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822413 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.822707 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.822681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.823088 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.823062    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.823190 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.823142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.823244 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.823224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.825344 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.825321    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.825484 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.825466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.879180 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.879147    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:28.904886 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:28.904859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:29.003204 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:29.003171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x" event={"ID":"120f0f3f-da9b-4395-807b-a291a37b5e2d","Type":"ContainerStarted","Data":"d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579"}
Oct 31 01:04:29.004489 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:29.004461    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh" event={"ID":"432f223d-4785-4a5c-af6a-921a65b1a08e","Type":"ContainerStarted","Data":"db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2"}
Oct 31 01:04:29.296005 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:29.295976    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5d58f5bd-vchll"]
Oct 31 01:04:29.327376 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:29.327330    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb40088_25e1_47b8_823a_f6ba2ef3e71d.slice/crio-aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544 WatchSource:0}: Error finding container aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544: Status 404 returned error can't find the container with id aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544
Oct 31 01:04:29.802365 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:29.802332    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:30.010112 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:30.010022    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d58f5bd-vchll" event={"ID":"7bb40088-25e1-47b8-823a-f6ba2ef3e71d","Type":"ContainerStarted","Data":"aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544"}
Oct 31 01:04:30.013516 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:30.013476    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"a44b1c73e240a41ed21de7ebe7278fe138dbadd35184bf982d2d9c3b0b550543"}
Oct 31 01:04:30.013712 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:30.013558    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"061a684c45a459c2fc14681bfdffbaff6c1de65679beabfc52ecfbf565ebb1cb"}
Oct 31 01:04:30.013712 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:30.013598    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"5e7e817c4d21c63f9da4038e7ad14fb38177999157211c3feffffce12cde7362"}
Oct 31 01:04:31.020824 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.020787    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"c9cba8ab3cc1a4bbdd574fc248eb4deb84034bf928f0c99020ec4b347bd06b9b"}
Oct 31 01:04:31.021260 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.020895    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"fa86480aca4f96ebde542863b8580260d1e3a4641d195c892e4e461a0eaff267"}
Oct 31 01:04:31.021260 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.020929    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" event={"ID":"911363c3-235f-4667-b12c-94089715f1bc","Type":"ContainerStarted","Data":"0e6cc1b752c74d81c438421ecf24043e8980d48219e1dd170a840fb80aa726f6"}
Oct 31 01:04:31.021260 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.020976    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:31.022846 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.022812    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x" event={"ID":"120f0f3f-da9b-4395-807b-a291a37b5e2d","Type":"ContainerStarted","Data":"4b5ed888c7fe423527e97abdda6442ce450ed4fcc6dfee67ca62b587aa257c87"}
Oct 31 01:04:31.023121 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.022875    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:31.025149 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.025067    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh" event={"ID":"432f223d-4785-4a5c-af6a-921a65b1a08e","Type":"ContainerStarted","Data":"015ff69749c0d60782e930d840a97ca5c306969b70e2cf73960fd23343cd1a7b"}
Oct 31 01:04:31.028864 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.028844    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:31.144852 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.144791    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6" podStartSLOduration=2.125849057 podStartE2EDuration="6.144770939s" totalImagesPullingTime="4.018921882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:31.143115426 +0000 UTC m=+59.927248379" watchObservedRunningTime="2026-10-31 01:04:31.144770939 +0000 UTC m=+59.928903878"
Oct 31 01:04:31.258613 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.258560    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x" podStartSLOduration=2.249997999 podStartE2EDuration="4.25854488s" totalImagesPullingTime="2.008546881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:31.257188111 +0000 UTC m=+60.041321051" watchObservedRunningTime="2026-10-31 01:04:31.25854488 +0000 UTC m=+60.042677863"
Oct 31 01:04:31.372631 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.372516    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh" podStartSLOduration=2.381442664 podStartE2EDuration="4.372497179s" totalImagesPullingTime="1.991054515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:31.371881861 +0000 UTC m=+60.156014802" watchObservedRunningTime="2026-10-31 01:04:31.372497179 +0000 UTC m=+60.156630120"
Oct 31 01:04:31.966589 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.966552    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:04:31.971604 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:31.971589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.029078 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.029038    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 01:04:32.029078 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.029052    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 01:04:32.055995 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.055959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056171 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056419 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056508 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056627 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056708 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056816 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056883 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.056931 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.056888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057044 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057125 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057308 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057415 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057483 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057592 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057592 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057713 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057713 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057869 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.057947 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.057881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.081668 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.080693    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 01:04:32.129476 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129445    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 01:04:32.129476 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129445    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 01:04:32.129476 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129443    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 01:04:32.129805 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129631    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zmtbq\""
Oct 31 01:04:32.129805 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129691    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsmab2l1uuhg0\""
Oct 31 01:04:32.129805 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129704    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 01:04:32.129953 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129923    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 01:04:32.129953 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.129945    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 01:04:32.130512 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.130480    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 01:04:32.158484 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158619 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158619 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158767 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158767 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158767 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.158918 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159060 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159060 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158942    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159060 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159060 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.158995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159060 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159298 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159640 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159620    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.159948 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.159899    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.160041 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.160003    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.160791 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.160180    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.161522 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.161470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.162512 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.162486    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.162610 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.162526    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.162764 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.162743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.162846 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.162765    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.163016 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.162997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.163259 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.163239    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.163343 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.163321    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.164215 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.164191    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.164410 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.164393    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.179384 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.179365    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 01:04:32.179646 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.179626    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 01:04:32.179746 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.179640    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 01:04:32.180025 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.180009    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 01:04:32.181416 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.181394    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:04:32.190225 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.190203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.192427 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.192377    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.192427 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.192397    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.192427 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.192407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.224873 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.224800    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.294573 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.294535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:32.780429 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.780396    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:04:32.783997 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:32.783958    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfe986f_2a26_4c74_9564_dbb4585d53cd.slice/crio-8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49 WatchSource:0}: Error finding container 8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49: Status 404 returned error can't find the container with id 8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49
Oct 31 01:04:32.968579 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:32.968539    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:33.035590 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:33.035500    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49"}
Oct 31 01:04:34.042216 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:34.042179    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d58f5bd-vchll" event={"ID":"7bb40088-25e1-47b8-823a-f6ba2ef3e71d","Type":"ContainerStarted","Data":"135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630"}
Oct 31 01:04:34.158543 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:34.158493    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b5d58f5bd-vchll" podStartSLOduration=1.799215158 podStartE2EDuration="6.158477519s" totalImagesPullingTime="4.359262361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:34.157443289 +0000 UTC m=+62.941576223" watchObservedRunningTime="2026-10-31 01:04:34.158477519 +0000 UTC m=+62.942610458"
Oct 31 01:04:35.047841 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:35.047810    2577 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="59b46adfead674c24e08869acb4255dd10bb6fdf2c26c6a0462c018cc49d1f80" exitCode=0
Oct 31 01:04:35.048304 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:35.047896    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"59b46adfead674c24e08869acb4255dd10bb6fdf2c26c6a0462c018cc49d1f80"}
Oct 31 01:04:37.037534 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.037504    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:37.508827 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.508787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:37.560083 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.560049    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:04:37.571929 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.571897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:37.609335 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.609299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:37.660936 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.660907    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:04:37.672593 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.672532    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:37.709944 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.709912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:37.761375 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.761281    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:04:37.778889 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.778864    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:04:37.788116 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.788088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:37.790079 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.790056    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:04:37.799464 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.799438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:37.822609 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.822581    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:04:37.834402 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:37.834350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:38.086111 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.086032    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:04:38.095265 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.095237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:38.184552 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.184530    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pf5kd"]
Oct 31 01:04:38.186962 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:38.186930    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e497701_d9b9_46b7_a5db_db294421d12f.slice/crio-5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c WatchSource:0}: Error finding container 5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c: Status 404 returned error can't find the container with id 5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c
Oct 31 01:04:38.239548 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.239368    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdsvq"]
Oct 31 01:04:38.241849 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:38.241817    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf64e4cb_1bd2_4a65_8d05_fb0d85be1f81.slice/crio-22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4 WatchSource:0}: Error finding container 22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4: Status 404 returned error can't find the container with id 22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4
Oct 31 01:04:38.480317 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.480241    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fczdh"]
Oct 31 01:04:38.484500 ip-10-0-141-86 kubenswrapper[2577]: W1031 01:04:38.484469    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792f66b0_e2d9_47c0_92bc_071ba29d52e5.slice/crio-13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989 WatchSource:0}: Error finding container 13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989: Status 404 returned error can't find the container with id 13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989
Oct 31 01:04:38.905801 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.905770    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:38.906541 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.906519    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:38.912494 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:38.912304    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:39.064668 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.064624    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdsvq" event={"ID":"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81","Type":"ContainerStarted","Data":"22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4"}
Oct 31 01:04:39.068102 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.068074    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"05d9c4bbca7496bc20f44a581d1afa8133d1113e2d8964905e993ce34d9feb3b"}
Oct 31 01:04:39.068276 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.068147    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"43e35f7873b5472f643e0f99b9dc03181d4715eb079cbfc948db13c14057ef72"}
Oct 31 01:04:39.068572 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.068542    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"e91dc4a49827f2eeb09a52cde1324c7d779092a8195c9e6f13a9ae99352a372b"}
Oct 31 01:04:39.068795 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.068777    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"f5fcf1217036a49bfc3bf12b3ec9e6f5e130c9250567c7045e1369bd4bf86486"}
Oct 31 01:04:39.068944 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.068930    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"72e0230c5bb4494cedbfc9fc957f9b4ff2c95c52336b4270e134e9ab8244a07d"}
Oct 31 01:04:39.069067 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.069055    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerStarted","Data":"7fbe6480290e3dc5d85c9552c1c9f9c5620a3b482c0998dc1f78ddf87ab7b930"}
Oct 31 01:04:39.070995 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.070959    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pf5kd" event={"ID":"2e497701-d9b9-46b7-a5db-db294421d12f","Type":"ContainerStarted","Data":"5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c"}
Oct 31 01:04:39.073803 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.073780    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fczdh" event={"ID":"792f66b0-e2d9-47c0-92bc-071ba29d52e5","Type":"ContainerStarted","Data":"13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989"}
Oct 31 01:04:39.077933 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.077891    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:39.204143 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:39.204093    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.279443534 podStartE2EDuration="8.204075555s" totalImagesPullingTime="4.910217638s" totalInitContainerRuntime="14.414383ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:39.202826843 +0000 UTC m=+67.986959783" watchObservedRunningTime="2026-10-31 01:04:39.204075555 +0000 UTC m=+67.988208495"
Oct 31 01:04:40.082093 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:40.081525    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdsvq" event={"ID":"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81","Type":"ContainerStarted","Data":"6f3c4d318543f4ff7f9485d0f8c2fcad6514e3435b3cbc8c9afaece3f95ac7f0"}
Oct 31 01:04:41.087824 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:41.087686    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdsvq" event={"ID":"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81","Type":"ContainerStarted","Data":"f8c2460ae1d23daf3c9faf945201661d66789350fc325a4d6486e2d318b7db3e"}
Oct 31 01:04:41.206748 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:41.205018    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rdsvq" podStartSLOduration=67.595784899 podStartE2EDuration="1m9.204996808s" totalImagesPullingTime="1.609211909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:41.20493123 +0000 UTC m=+69.989064170" watchObservedRunningTime="2026-10-31 01:04:41.204996808 +0000 UTC m=+69.989129741"
Oct 31 01:04:42.294775 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:42.294744    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:43.098334 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:43.098297    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fczdh" event={"ID":"792f66b0-e2d9-47c0-92bc-071ba29d52e5","Type":"ContainerStarted","Data":"0d083c93e7879b45ea8a47a547b08e70d4aa43afeba603fda51075c72c501671"}
Oct 31 01:04:43.098525 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:43.098445    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:43.220093 ip-10-0-141-86 kubenswrapper[2577]: I1031 01:04:43.220024    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fczdh" podStartSLOduration=67.471527507 podStartE2EDuration="1m11.220004621s" totalImagesPullingTime="3.748477114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:43.219215166 +0000 UTC m=+72.003348104" watchObservedRunningTime="2026-10-31 01:04:43.220004621 +0000 UTC m=+72.004137561"
Oct 31 01:04:43.769714 ip-10-0-141-86 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 01:04:43.779374 ip-10-0-141-86 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 01:04:43.779610 ip-10-0-141-86 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 01:04:43.779698 ip-10-0-141-86 systemd[1]: kubelet.service: Consumed 4.598s CPU time.
Oct 31 01:04:43.792281 ip-10-0-141-86 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:04:43.875230 ip-10-0-141-86 kubenswrapper[5718]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:04:43.875230 ip-10-0-141-86 kubenswrapper[5718]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:04:43.875230 ip-10-0-141-86 kubenswrapper[5718]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:04:43.875230 ip-10-0-141-86 kubenswrapper[5718]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:04:43.877498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877399    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.877498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877497    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.877498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877502    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877506    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877509    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877512    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877515    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877524    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877527    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877530    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877533    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877536    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877538    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877541    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877544    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877546    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877548    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877551    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877553    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877556    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877558    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877561    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.877639 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877563    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877566    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877568    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877571    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877574    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877576    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877581    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877584    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877586    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877589    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877592    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877594    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877597    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877600    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877603    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877605    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877608    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877613    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877616    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877619    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.878132 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877621    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877624    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877626    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877629    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877631    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877634    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877636    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877639    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877641    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877644    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877647    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877678    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877682    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877687    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877690    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877700    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877703    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877706    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877708    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877712    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.878617 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877715    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877718    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877721    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877723    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877726    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877728    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877731    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877733    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877736    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877739    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877745    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877748    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877751    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877754    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877756    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877759    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877761    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877764    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877767    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877769    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.879161 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877772    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877774    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877777    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877781    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877783    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877788    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877792    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877795    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877798    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877801    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877803    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877806    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877808    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877811    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877814    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877816    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877818    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877824    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877828    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.879641 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877831    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877834    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877836    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877839    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.877842    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878676    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878687    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878690    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878694    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878697    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878700    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878702    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878705    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878708    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878711    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878714    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878717    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878719    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878723    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.880165 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878727    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878731    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878736    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878741    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878744    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878748    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878753    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878757    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878761    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878764    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878766    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878770    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878775    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878778    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878781    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878784    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878787    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878789    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878793    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.880712 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878797    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878803    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878807    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878812    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878816    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878819    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878821    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878824    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878827    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878829    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878832    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878835    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878837    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878840    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878842    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878845    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878847    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878850    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878852    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878855    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.881178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878857    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878860    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878863    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878865    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878868    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878870    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878872    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878875    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878877    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878880    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878882    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878885    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878888    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878891    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878894    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878896    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878899    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878901    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878904    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.881685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878906    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878908    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878911    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878914    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878916    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878919    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878922    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878924    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878927    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878929    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878932    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878934    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878937    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878939    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878942    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878944    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878947    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878949    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878951    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.882193 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878954    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878956    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878959    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878961    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878963    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878966    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878969    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878972    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878976    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878978    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878983    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878987    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878990    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878993    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.878996    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879518    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879524    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879527    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879530    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879532    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.882679 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879535    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879538    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879540    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879543    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879545    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879548    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879551    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879553    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879556    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879558    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879560    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879563    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879565    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879569    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879571    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879575    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879577    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879580    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879583    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.883174 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879586    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879589    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879591    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879594    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879597    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879599    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879602    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879604    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879607    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879610    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879612    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879615    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879617    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879619    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879622    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879625    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879627    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879631    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879635    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.883664 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879638    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879641    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879644    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879665    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879669    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879672    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879675    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879677    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879680    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879683    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879686    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879688    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879691    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879693    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879697    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879699    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879702    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879704    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879707    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.884182 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879709    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879712    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879714    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879716    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879719    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879721    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879724    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879728    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879737    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879743    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879748    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879751    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879754    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879757    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879759    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879762    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879765    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879767    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879770    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.884685 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879773    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879775    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879777    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879780    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879783    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879788    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879790    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879793    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879796    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879800    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879802    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879805    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879807    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879809    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879812    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879815    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879817    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879820    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879822    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879825    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.885145 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879828    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879832    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879835    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879839    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.879841    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879947    5718 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879955    5718 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879964    5718 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879969    5718 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879974    5718 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879977    5718 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879982    5718 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879986    5718 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879989    5718 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879992    5718 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879996    5718 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.879999    5718 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880003    5718 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880006    5718 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880009    5718 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880012    5718 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880015    5718 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880019    5718 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880022    5718 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:04:43.885645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880025    5718 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880028    5718 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880031    5718 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880035    5718 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880039    5718 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880042    5718 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880045    5718 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880048    5718 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880051    5718 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880056    5718 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880060    5718 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880063    5718 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880066    5718 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880071    5718 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880074    5718 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880079    5718 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880082    5718 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880085    5718 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880088    5718 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880125    5718 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880128    5718 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880132    5718 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880135    5718 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880138    5718 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:04:43.886267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880144    5718 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880147    5718 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880151    5718 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880154    5718 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880157    5718 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880160    5718 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880164    5718 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880168    5718 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880171    5718 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880175    5718 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880178    5718 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880183    5718 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880186    5718 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880189    5718 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880193    5718 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880196    5718 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880200    5718 flags.go:64] FLAG: --help="false"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880204    5718 flags.go:64] FLAG: --hostname-override="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880208    5718 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880211    5718 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880214    5718 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880217    5718 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880221    5718 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:04:43.887081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880225    5718 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880229    5718 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880232    5718 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880234    5718 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880237    5718 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880241    5718 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880244    5718 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880248    5718 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880250    5718 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880254    5718 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880256    5718 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880259    5718 flags.go:64] FLAG: --lock-file=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880262    5718 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880265    5718 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880269    5718 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880274    5718 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880277    5718 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880281    5718 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880285    5718 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880287    5718 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880291    5718 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880294    5718 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880297    5718 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880309    5718 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880312    5718 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:04:43.887902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880316    5718 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880320    5718 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880324    5718 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880327    5718 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880330    5718 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880333    5718 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880336    5718 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880342    5718 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880351    5718 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880354    5718 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880357    5718 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880361    5718 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880364    5718 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880368    5718 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880371    5718 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880374    5718 flags.go:64] FLAG: --port="10250"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880377    5718 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880380    5718 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0cd84a8000fadabfa"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880383    5718 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880386    5718 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880390    5718 flags.go:64] FLAG: --register-node="true"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880392    5718 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880396    5718 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880400    5718 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880402    5718 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880417    5718 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:04:43.888505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880422    5718 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880426    5718 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880429    5718 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880433    5718 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880436    5718 flags.go:64] FLAG: --runonce="false"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880439    5718 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880443    5718 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880447    5718 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880450    5718 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880453    5718 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880457    5718 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880460    5718 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880463    5718 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880467    5718 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880470    5718 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880473    5718 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880476    5718 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880480    5718 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880483    5718 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880485    5718 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880491    5718 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880494    5718 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880497    5718 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880500    5718 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880503    5718 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:04:43.889181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880506    5718 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880509    5718 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880511    5718 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880514    5718 flags.go:64] FLAG: --v="2"
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880519    5718 flags.go:64] FLAG: --version="false"
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880524    5718 flags.go:64] FLAG: --vmodule=""
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880528    5718 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.880531    5718 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880707    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880713    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880716    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880719    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880723    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880726    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880730    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880732    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880736    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880739    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880741    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880744    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880748    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.889808 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880751    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880754    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880756    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880759    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880762    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880765    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880768    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880770    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880773    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880775    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880777    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880780    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880782    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880785    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880787    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880790    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880793    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880796    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880798    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880802    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.890326 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880804    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880807    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880809    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880812    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880814    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880818    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880821    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880824    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880826    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880829    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880831    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880836    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880839    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880841    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880844    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880847    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880851    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880855    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880858    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.890936 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880861    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880864    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880866    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880869    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880871    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880874    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880876    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880879    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880881    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880884    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880887    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880889    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880892    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880896    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880898    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880901    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880903    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880906    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880910    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.891399 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880912    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880914    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880917    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880920    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880923    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880927    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880930    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880933    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880936    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880938    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880941    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880943    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880946    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880948    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880953    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880956    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880959    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880962    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880965    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880968    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.891930 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880970    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880973    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880976    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880978    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880981    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880983    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880986    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880989    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880991    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880994    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.880996    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881000    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881002    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881005    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881007    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.881038    5718 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:04:43.892443 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881086    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881093    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881097    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881102    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881105    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881107    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881110    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881112    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881115    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881118    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881121    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881123    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881126    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881128    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881131    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881133    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881136    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881139    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881141    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881144    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.893037 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881146    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881149    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881152    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881154    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881157    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881160    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881162    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881165    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881168    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881171    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881173    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881176    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881178    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881182    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881185    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881188    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881190    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881193    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881196    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.893796 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881198    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881201    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881203    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881206    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881208    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881211    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881213    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881216    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881218    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881221    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881223    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881226    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881228    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881231    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881233    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881236    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881238    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881240    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881243    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881246    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.894349 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881248    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881252    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881254    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881257    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881259    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881261    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881265    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881268    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881270    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881272    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881276    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881279    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881282    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881286    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881289    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881292    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881295    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881297    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881300    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.895114 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881303    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881305    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881307    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881310    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881312    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881314    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881317    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881320    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881322    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881325    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881327    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881330    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881333    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881335    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881338    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881340    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881342    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881345    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881347    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.895878 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881351    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881354    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881356    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881358    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881379    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881383    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881386    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881389    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.881391    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.885872    5718 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.885899    5718 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.885918    5718 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.885924    5718 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.885987    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.885994    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.885998    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886001    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886004    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886006    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.896566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886011    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886016    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886019    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886022    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886024    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886027    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886030    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886032    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886035    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886038    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886040    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886043    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886045    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886048    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886051    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886054    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886056    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886059    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886061    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886064    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.897388 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886067    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886070    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886073    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886076    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886079    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886082    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886085    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886089    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886092    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886095    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886097    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886100    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886102    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886105    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886107    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886110    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886112    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886115    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886117    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.897929 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886120    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886122    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886125    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886127    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886130    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886133    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886136    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886139    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886141    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886144    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886147    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886149    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886152    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886155    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886157    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886160    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886162    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886165    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886168    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886170    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.898498 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886173    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886176    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886178    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886181    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886184    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886186    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886189    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886191    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886194    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886197    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886200    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886202    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886205    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886207    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886210    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886212    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886215    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886217    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886220    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.899289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886223    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886225    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886228    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886230    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886233    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886235    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886238    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886240    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886243    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886246    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886248    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886251    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886253    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886256    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886259    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886261    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886264    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886266    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886269    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886272    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.899770 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886275    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886277    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.886313    5718 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.886938    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887190    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887195    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887201    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887205    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887210    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887222    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887226    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887230    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887235    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887240    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887244    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887249    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.900442 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887256    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887261    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887265    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887275    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887282    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887292    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887297    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887302    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887308    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887313    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887317    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887322    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887327    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887331    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887336    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887340    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887345    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887349    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887359    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.900861 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887363    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887367    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887372    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887376    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887380    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887384    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887389    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887394    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887398    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887402    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887407    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887415    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887420    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887424    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887429    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887435    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887439    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887444    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887448    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887453    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.901324 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887457    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887462    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887467    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887471    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887480    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887484    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887489    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887493    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887497    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887502    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887507    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887511    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887516    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887520    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887525    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887530    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887539    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887543    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887547    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887551    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.901911 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887555    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887559    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887563    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887568    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887572    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887576    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887580    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887584    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887588    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887597    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887604    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887609    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887613    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887618    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887622    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887626    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887630    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887640    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887644    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.902539 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887675    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887679    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887682    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887686    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887690    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887695    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887698    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887701    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887704    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887707    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887714    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887719    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887723    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887727    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.887732    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.887972    5718 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:04:43.903248 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888029    5718 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888035    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888038    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888041    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888044    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888047    5718 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888051    5718 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888054    5718 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888058    5718 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888061    5718 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888064    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888067    5718 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888071    5718 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888073    5718 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888076    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888079    5718 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888082    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888085    5718 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888087    5718 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888090    5718 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:43.903876 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888092    5718 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888095    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888098    5718 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888100    5718 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888103    5718 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888105    5718 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888109    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888112    5718 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888114    5718 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888117    5718 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888119    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888122    5718 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888125    5718 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888127    5718 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888130    5718 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888133    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888135    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888138    5718 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888141    5718 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:43.904540 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888144    5718 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888146    5718 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888148    5718 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888151    5718 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888154    5718 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888156    5718 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888160    5718 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888164    5718 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888167    5718 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888170    5718 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888172    5718 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888175    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888178    5718 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888180    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888183    5718 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888185    5718 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888188    5718 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888191    5718 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888193    5718 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888196    5718 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:43.905259 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888199    5718 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888202    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888204    5718 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888207    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888209    5718 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888212    5718 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888218    5718 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888221    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888223    5718 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888226    5718 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888230    5718 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888233    5718 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888236    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888240    5718 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888244    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888247    5718 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888260    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888264    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888267    5718 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:43.905823 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888269    5718 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888272    5718 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888274    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888277    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888280    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888283    5718 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888285    5718 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888288    5718 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888290    5718 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888293    5718 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888295    5718 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888298    5718 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888300    5718 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888303    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888306    5718 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888308    5718 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888311    5718 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888314    5718 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888316    5718 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888322    5718 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:43.906288 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888325    5718 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888327    5718 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888330    5718 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888332    5718 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888335    5718 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888338    5718 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888340    5718 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:04:43.888343    5718 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.888441    5718 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.889564    5718 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.889638    5718 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.889874    5718 server.go:1014] "Starting client certificate rotation"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.890011    5718 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.890073    5718 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:58:31 +0000 UTC" deadline="2028-05-29 11:55:48.34281865 +0000 UTC"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.890103    5718 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13834h51m4.452717641s"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.890398    5718 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:04:43.906801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.891793    5718 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:04:43.907225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.893212    5718 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:04:43.907225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.897443    5718 log.go:25] "Validated CRI v1 image API"
Oct 31 01:04:43.907225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.898573    5718 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:04:43.907225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.901118    5718 fs.go:135] Filesystem UUIDs: map[21439fdd-9ff9-460d-8fda-3306979c2c32:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f32ddd08-39f2-4ca2-9d5b-2c0526f294af:/dev/nvme0n1p3]
Oct 31 01:04:43.907348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.901137    5718 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4/userdata/shm major:0 minor:414 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~projected/kube-api-access-x84vl:{mountpoint:/var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~projected/kube-api-access-x84vl major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/120f0f3f-da9b-4395-807b-a291a37b5e2d/volumes/kubernetes.io~projected/kube-api-access-bkf97:{mountpoint:/var/lib/kubelet/pods/120f0f3f-da9b-4395-807b-a291a37b5e2d/volumes/kubernetes.io~projected/kube-api-access-bkf97 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/120f0f3f-da9b-4395-807b-a291a37b5e2d/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/120f0f3f-da9b-4395-807b-a291a37b5e2d/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1608abd9-fd52-44e3-a945-eccf76f178aa/volumes/kubernetes.io~projected/kube-api-access-29d76:{mountpoint:/var/lib/kubelet/pods/1608abd9-fd52-44e3-a945-eccf76f178aa/volumes/kubernetes.io~projected/kube-api-access-29d76 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1608abd9-fd52-44e3-a945-eccf76f178aa/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1608abd9-fd52-44e3-a945-eccf76f178aa/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e8405b7-4f76-495c-b9fc-04affdf8d1f8/volumes/kubernetes.io~projected/kube-api-access-pc2r4:{mountpoint:/var/lib/kubelet/pods/1e8405b7-4f76-495c-b9fc-04affdf8d1f8/volumes/kubernetes.io~projected/kube-api-access-pc2r4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a09fe47-166b-4fa0-9982-afffd73320b8/volumes/kubernetes.io~projected/kube-api-access-2xjhl:{mountpoint:/var/lib/kubelet/pods/2a09fe47-166b-4fa0-9982-afffd73320b8/volumes/kubernetes.io~projected/kube-api-access-2xjhl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volumes/kubernetes.io~projected/kube-api-access-jxmqv:{mountpoint:/var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volumes/kubernetes.io~projected/kube-api-access-jxmqv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e497701-d9b9-46b7-a5db-db294421d12f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2e497701-d9b9-46b7-a5db-db294421d12f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~projected/kube-api-access-gz8n7:{mountpoint:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~projected/kube-api-access-gz8n7 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373/volumes/kubernetes.io~projected/kube-api-access-q9xcq:{mountpoint:/var/lib/kubelet/pods/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373/volumes/kubernetes.io~projected/kube-api-access-q9xcq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~projected/kube-api-access-6r9jn:{mountpoint:/var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~projected/kube-api-access-6r9jn major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/792f66b0-e2d9-47c0-92bc-071ba29d52e5/volumes/kubernetes.io~projected/kube-api-access-jmslr:{mountpoint:/var/lib/kubelet/pods/792f66b0-e2d9-47c0-92bc-071ba29d52e5/volumes/kubernetes.io~projected/kube-api-access-jmslr major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~projected/kube-api-access-rctgq:{mountpoint:/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~projected/kube-api-access-rctgq major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~projected/kube-api-access-769kz:{mountpoint:/var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~projected/kube-api-access-769kz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~projected/kube-api-access-mmx7m:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~projected/kube-api-access-mmx7m major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~projected/kube-api-access-xpn68:{mountpoint:/var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~projected/kube-api-access-xpn68 major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81/volumes/kubernetes.io~projected/kube-api-access-2g25n:{mountpoint:/var/lib/kubelet/pods/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81/volumes/kubernetes.io~projected/kube-api-access-2g25n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81/volumes/kubernetes.io~secret/metrics-certs major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603/volumes/kubernetes.io~projected/kube-api-access-w2qs9:{mountpoint:/var/lib/kubelet/pods/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603/volumes/kubernetes.io~projected/kube-api-access-w2qs9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa76e03-b876-4bc5-a0ec-aab3cced55f0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/caa76e03-b876-4bc5-a0ec-aab3cced55f0/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/config-reloader/2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/prometheus/15 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~empty-dir/config-out major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~projected/kube-api-access-gw6tt:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~projected/kube-api-access-gw6tt major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~projected/tls-assets major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/config major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/web-config major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de7aadea-e0a6-4086-ae7a-a8f7794ad264/volumes/kubernetes.io~projected/kube-api-access-gn884:{mountpoint:/var/lib/kubelet/pods/de7aadea-e0a6-4086-ae7a-a8f7794ad264/volumes/kubernetes.io~projected/kube-api-access-gn884 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9562f13-b930-4947-9aac-9379535e8fe6/volumes/kubernetes.io~projected/kube-api-access-p46tq:{mountpoint:/var/lib/kubelet/pods/e9562f13-b930-4947-9aac-9379535e8fe6/volumes/kubernetes.io~projected/kube-api-access-p46tq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051a847-0102-43c1-8fd4-7885e50459c2/volumes/kubernetes.io~projected/kube-api-access-kbtpv:{mountpoint:/var/lib/kubelet/pods/f051a847-0102-43c1-8fd4-7885e50459c2/volumes/kubernetes.io~projected/kube-api-access-kbtpv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051a847-0102-43c1-8fd4-7885e50459c2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f051a847-0102-43c1-8fd4-7885e50459c2/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb29d15c-7076-454a-a335-22e9b8097c4f/volumes/kubernetes.io~projected/kube-api-access-f5lkz:{mountpoint:/var/lib/kubelet/pods/fb29d15c-7076-454a-a335-22e9b8097c4f/volumes/kubernetes.io~projected/kube-api-access-f5lkz major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a39bdb51e4898e82b677306c0d6bf101b33b48cbe6f0f2652f6f76a260755dbb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/efa778c9b5a36eefa0c58c473d6d3eff65da068ac0a53119c0f44c257cc7699c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/45a934f93a3467808e30cd5b34f9b05f778a5dc4cd45806f310c2e9bd20b4cc4/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/883fc6a4c024b379a5a774ef3106bcc25e3805abd1187c51015b123b1ebcff8b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4dba15c31072668d07f8f3ce738c99c586505d8849a18922803373ef6eaddb69/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d8ff16890a5bcdd6a88a0a34ae3f034ea0c82227bad4ceb63975759aa9dccb27/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/84b5b1d6b043c64a2db820f4bf1f28f5a122031082f809885fb991cc52b90341/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c67f41101cffaf909f908151f083f304b0550501bbf393498c79ce953aeff623/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/51580005c9d55339da2bcb02260fcaa54c98cf16e46f1a439dec17c566619478/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8acd9c264063c8b1414040c9b956d271f0c4147a3b57486f7a28954084a4b054/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/6790dd0d004e0eabccecfdf92b882be5f18d61408dab7eb66d2fedce50cde8d8/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/68917b5129510dc03864f15bbc21375413eb2b1aedd03bbd39305d591694da2e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e3cab337ca7da672d736977bddca0dcb74548a03f5c203526e5fa4be15a5fbf6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fa9d7a5006e4abedaa2aa16bde97f61e0852c08b95a598fb203a3df95e7357ba/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/487e70570cc2081cc1b60d3183032b1a8e70320049070c5116ef19aabcf5092c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/affd38194878f46efc4734d1c39c7303f9cc94d71bfbbb40284ccaa98a170942/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/61d812abc00f2a5316a2c190c93f80c96462f5750404785b1dc6ca84718959a0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d6310ab481292f53063b004be4f157f529e8421de5de240f47313d70cf5cb909/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/799bc2a7a50d6fd78c581ff494a6cc7ed20a5f3bdb5963b2e21503debef891b3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e89bcbe0b7b5013c41b408eb9c710da80d5f11bdd5de59499fbcc9a96990e234/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3cd365205121fa2df6309bc0b1bd35e868ff667d82bc4847d7d71e0dda35bfb1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ecc5d99a89e6dd8ea8b74fad70f3be0c044fb9f44e153720e1ae2e5b1e451f74/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/7645d352473d5c2c0dd9c38bce4b46ff45f22e4318d4b0c2dd77c2ccacbe0792/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f73608619e64ee96bf366c98e59b0c3bbbd086d9974af25b5586b897a44a70c8/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/19653928ce0b20fbf9cb88fec0a84931322254bb924dd3a628ca4a6f552b6bce/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/43dd1f86be2adb447af971269075f6d8d768e44cfde12e49de16b6d024a7e03a/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/001ac627ca0226e4273d9081865fd03ef0c1436e265d44181e28a9c2b82e1e59/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/83cd215a4e4645652122835605379b74aafb5aed214ec7cd40443a464dac975f/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/d0eb6cc127b3279ae5432e25696024d69aaf39e0f6e73a11cc81a18e8ae34fed/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/86a352c96c6707c6a3a097b76fe3465c644247693d2644e38a69094ed96f31f8/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/bfdb3dab615a91d4a966905d59c73ffd358ef357d611eeca63d255f6cd378d9e/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/9f527c4213f3e0007b1a981913b89c07edb06a1480ee8bc504233d57e8d4bd4d/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/6ec01ae3a7a0271a5c71ce30727df296ce06a46a2aca78828e009493b2f86fd4/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/f2e421ea3ab650ca9399d0db3b528ab49a09b2c3b2b9045b0297002388094bc2/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/1714931bf22bc6e08f775f379f991fae8c280d5419da25e00ba67be99b791953/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/5cbde8a5614673d60eaf4d02533e1bd6b0872ce47d07f96e1610bb2a3a233b7e/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/23a977f6534add01e2e597cf018bcc70dd176e1faa49c55ae2d90194bd15a55a/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/a9d46c9eb9f6dcfae6e32212f2b79249b5a6d8c6690cf517b605f03d6f26c11c/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/170f9a4511d5b55eb1856b92a2bf13a495e61d6b103337ec2e93d0a5833a2eeb/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/2d8677946faba88a12ffa145dac1c8df36a285c71440a23b73a17d6cd143d99a/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/6b4c4bc674166cdad2310e0cca1d90ebbf521ebfb66e000e29fa6a9989c6c8e7/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/fd2c1da90536a4d8be8ac445b121b0c93e3d28622bd4bd301aca62842508da0f/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/69a133b81716094106ec3e202c03c758f881e8b31564a9986007bd74b7893a15/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/23235827b59862b6bc1e2afc51153fb4441cb1b1978a1d8fca17923813837432/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/574123a8e4090f57fcf9d3dcfa5657734945762518094d4df1825323c907181f/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/2b1352ab4774648df2a528c8c9ff68b432c55f51f091028032d0972bd6fd09f0/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/0392bfc05d840b414e378a4ecc7974956098117d416f09c37b77dd50553fda1a/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/3d91c8ab3e16f40dbf7c80c079ebde6519f44c2e73501cd9eff1d21e1c53617f/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/04e9ff1bfe8a2e9a328e5ee2a2701bb9fcad42fde8c937556bc08795c0d96269/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/e74df586bb5ca988456ff97e59470e6532200b60ff3b8e013633359ab2106647/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/ebf56fdb7e87f809b9df62f8738e11c80bfdb2c9b87a852e918d2c18975dd23b/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/97c0d3f44f0a10d9daad46c48519e7bcc25023616ce6690f8fd6ff205db50f8b/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/d2e52a0d0e757bc664ca9ed6015b844517a46b66278bf25182612ff63f06504a/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/26f07b37ebbdb04b441b7fbd8986d19d2d0e6fec6ae0334429927a08c9295262/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/95ab2257c7baf8ac10e6e716b07efae010ada6af40308201f0d341f41e602f84/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/21a49b41ad1321e9ed3b0e143b20f9bfc7d05877740f3078673082a91a22a1ca/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/90937f1ad6c2348c50d9282aac303ec69a918ea7aedaa682efe24e212e677171/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/f3d94f1aea6c3c5d0fbd2be25f24866fd101d7a30009e5908ef2c7a46494abdd/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/6faa9da18615f572da2c7de8d5699280ad1c011f02d9869c10f5a2a5bfe61a6f/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/01fa4d759a0c5bbd63d1ea3956a102fb80b00f922f395c061395993802bc42d2/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/fa531b276cf009410e355de8b270ac0f105e6b8e118e5b1ff297b6cc3785945e/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/12f53376ea5685edb1d3b599d27de0771d390e0e4e9776955dce4a212d8ffe84/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/e5821a94768b3b29003359b65e28c55322599554992db65282f7d28b5dfb4c34/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/be0d9c1fdbc1d41c2fcb4442f73762c7c25a30967aca29064b2c1f36fc20aac4/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/2b5caec8115e1b552329e7236031607ecf13169554173559412356cdbd701d29/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/61a09c42ac7f2f94bc20a247c5ddf072029dd9690490fa048b3447c39a6beaee/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/ad7dce016fe7ffd01e858d320566513b6a8cb99829fb5a64c11747fb178e3d04/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/008785a861d96cfef48ec14a6c21e17d048c0bfba869abd42616d15c9ad8c056/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9da13f401a66679f8db94f8f8dc6b2e2061b760e4e48162cb7b0481636fdafa6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e4bd44ad2abb9e37406ea3237e79b40091fb16772b2c53f4abe725f9216e47af/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/2dc31fd2691ba0077677ecbf175466f56436985606329b3dee79bbf935de158f/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5a9b2f54eb2edef75cb6646c385bf46b014426d70e6a0e68550ab6600ca1c8e0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e94b7cc541ee064e52d48e4824838b28c9123a59b089715178bcd20e19d4843c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/eaede3720c40bd2ca822b9f0cc4baf2527939ca1a700038fda5c37f8b4f03a9e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a670ce47b40a3c7b46159aa5969cfcdd32da35df1b5b24c830d11dc4cc46d291/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/71a69842df828da251a5f5c0128c0d582baf4a739f5ac308a1e67b1c9d6b430e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6d94dbeae5e5e190de43034b9b5c2f25d753ad2ae0f1485861b6a1898e33ffc3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/02baa7f11b0d6cfee96dc9dad0390eecb3baae4b77164f192040489907df9c49/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 01:04:43.909326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.908866    5718 manager.go:217] Machine: {Timestamp:2026-10-31 01:04:43.908360371 +0000 UTC m=+0.071503777 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100333 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2500774b2f03d360526cd57e9f7270 SystemUUID:ec250077-4b2f-03d3-6052-6cd57e9f7270 BootID:93c51d36-cdef-4404-836e-79b83c32e53e Filesystems:[{Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9562f13-b930-4947-9aac-9379535e8fe6/volumes/kubernetes.io~projected/kube-api-access-p46tq DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/de7aadea-e0a6-4086-ae7a-a8f7794ad264/volumes/kubernetes.io~projected/kube-api-access-gn884 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e44a76253ae42849bd9f43f2253b685001d4a261231b57a0c82c3a15a4e64fa6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e8405b7-4f76-495c-b9fc-04affdf8d1f8/volumes/kubernetes.io~projected/kube-api-access-pc2r4 DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1608abd9-fd52-44e3-a945-eccf76f178aa/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1608abd9-fd52-44e3-a945-eccf76f178aa/volumes/kubernetes.io~projected/kube-api-access-29d76 DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373/volumes/kubernetes.io~projected/kube-api-access-q9xcq DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db71e2413803231f45f4a55b3e00dfa8ff152cd57114663d11f4fa5abbc85ca2/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:381 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0f7bf9d0c8194b2894afd220f9f745c8a46c9b6e4ffb80a0a8ce3c1645b1997/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:381 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db37d031fb7e45e70ff279220b2c3c315794792d5ee7851afdd7d28c6c5c5d89/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4b0c5d3a85bb6f4ac64571e0adf7a467ae9fcb405c1d7077282ecdfa57fdd2f/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/792f66b0-e2d9-47c0-92bc-071ba29d52e5/volumes/kubernetes.io~projected/kube-api-access-jmslr DeviceMajor:0 DeviceMinor:416 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:330 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cb721a7ea9afd5cf1a52e28520de4bc2f67f30009043f1bd20a0c9ca604ce8c/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73b1d62b6a72c5ceeead365048d121593ef41eae29d372ddfbb104ae45e3cca7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c070fd841545a25d1b2f4fc82efd583f06af5b4fc20ef751786738bbf6bad440/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea88e1c59bb1d3276ad71101cb4b376cc12e093c5aeefce64a6d3ce766cc8a1e/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed6013dab8933d7baa85e112af316896d4026896c9286adc3c0151b64e12e261/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/120f0f3f-da9b-4395-807b-a291a37b5e2d/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:324 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81/volumes/kubernetes.io~projected/kube-api-access-2g25n DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volumes/kubernetes.io~projected/kube-api-access-jxmqv DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:294 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:299 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:300 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:393 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0253f3f6f68238f59451de7b7e6e9655950c5154c8c9e44a43e5730f4ac9be35/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~projected/kube-api-access-mmx7m DeviceMajor:0 DeviceMinor:310 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:396 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:406 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:256 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~projected/kube-api-access-6r9jn DeviceMajor:0 DeviceMinor:252 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:381 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6d1e5640920de3f58523845c27b33c905a8b4aa7dcae6fadc18dda64737c579/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13958cc828154f811358d25d86cc503c0fa184f973b70b43493c334bd2539989/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/fb29d15c-7076-454a-a335-22e9b8097c4f/volumes/kubernetes.io~projected/kube-api-access-f5lkz DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:382 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5a2fca918ae910ad83fb81ca726e74b692bedb97c6bc4508efd17b7f5859d88/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:395 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~projected/kube-api-access-769kz DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:328 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:394 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de4b1594b981a037dad34f0327689254978a925e94ffd25ecdebf5f54abc69c3/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:298 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:301 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:381 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/911363c3-235f-4667-b12c-94089715f1bc/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/caa76e03-b876-4bc5-a0ec-aab3cced55f0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603/volumes/kubernetes.io~projected/kube-api-access-w2qs9 DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/297e8338bfec7d27f53df29c4bc61a2729e56e5fdab14e6ec9f74f9ecc5b6c20/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~projected/kube-api-access-gz8n7 DeviceMajor:0 DeviceMinor:331 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/05380ea7-efbf-4c24-9163-97a4243365df/volumes/kubernetes.io~projected/kube-api-access-x84vl DeviceMajor:0 DeviceMinor:258 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:390 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:397 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e497701-d9b9-46b7-a5db-db294421d12f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:411 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3cee94-19dc-4e56-b7a2-256765111acb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/961e1a63-66d7-4c5f-924e-27d9d56d6862/volumes/kubernetes.io~projected/kube-api-access-xpn68 DeviceMajor:0 DeviceMinor:257 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:392 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54928e9319fe3fb028b38ab61923f0050d9a32ee175687b1abda27db174c64f0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/432f223d-4785-4a5c-af6a-921a65b1a08e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c464f6d-9424-478f-8b7c-c64b564d816f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051a847-0102-43c1-8fd4-7885e50459c2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3be85456a195af813d823d53b8e27a76c6c79f7201e830135da3dd20af5e20d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/65337868-194c-4e3f-9e3f-7e96c35c6761/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:398 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22ecc80c83f3d63c7b880fa3bae7578f9e4fead09527f9f85ee277a6ddb1d9d4/userdata/shm DeviceMajor:0 DeviceMinor:414 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051a847-0102-43c1-8fd4-7885e50459c2/volumes/kubernetes.io~projected/kube-api-access-kbtpv DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd8bc7f69de52782770865eeea68f9c681224f068076aced0a755944e8cbd05c/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:338 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~projected/kube-api-access-rctgq DeviceMajor:0 DeviceMinor:340 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a09fe47-166b-4fa0-9982-afffd73320b8/volumes/kubernetes.io~projected/kube-api-access-2xjhl DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/120f0f3f-da9b-4395-807b-a291a37b5e2d/volumes/kubernetes.io~projected/kube-api-access-bkf97 DeviceMajor:0 DeviceMinor:325 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:391 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes/kubernetes.io~projected/kube-api-access-gw6tt DeviceMajor:0 DeviceMinor:399 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:13958cc828154f8 MacAddress:d2:e2:46:c0:6a:6c Speed:10000 Mtu:8901} {Name:22ecc80c83f3d63 MacAddress:ba:d3:6e:44:26:97 Speed:10000 Mtu:8901} {Name:5cb721a7ea9afd5 MacAddress:62:cb:5d:a9:2b:cf Speed:10000 Mtu:8901} {Name:8a701991fc0e1dc MacAddress:de:77:44:ed:a0:3d Speed:10000 Mtu:8901} {Name:aa942a4855493ac MacAddress:5e:d0:3f:df:82:c3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:32:86:98:70:cb Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:83:c9:19:ff:94 Speed:0 Mtu:8901} {Name:c070fd841545a25 MacAddress:de:87:ac:42:56:50 Speed:10000 Mtu:8901} {Name:c3be85456a195af MacAddress:de:c1:0a:6b:da:54 Speed:10000 Mtu:8901} {Name:d6d1e5640920de3 MacAddress:36:b5:2b:8b:a7:58 Speed:10000 Mtu:8901} {Name:db71e2413803231 MacAddress:42:f7:b9:aa:94:b3 Speed:10000 Mtu:8901} {Name:de4b1594b981a03 MacAddress:c6:b5:19:70:94:e6 Speed:10000 Mtu:8901} {Name:ea88e1c59bb1d32 MacAddress:42:b3:33:21:11:29 Speed:10000 Mtu:8901} {Name:ed6013dab8933d7 MacAddress:0a:1c:7e:60:6e:90 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:32:86:98:70:cb Speed:0 Mtu:9001} {Name:f5a2fca918ae910 MacAddress:ba:b3:fd:1a:b7:be Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1a:2b:f0:af:a2:58 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:8d:67:d1:41:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:04:43.909326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.909331    5718 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:04:43.910043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.909386    5718 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:04:43.910138 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910101    5718 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:04:43.910444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910143    5718 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-86.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:04:43.910568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910450    5718 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:04:43.910568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910465    5718 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:04:43.910568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910482    5718 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:04:43.910568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910510    5718 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:04:43.910746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910698    5718 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:04:43.910970 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.910954    5718 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:04:43.911146 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.911134    5718 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:04:43.911201 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.911183    5718 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:04:43.911246 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.911206    5718 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:04:43.911246 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.911225    5718 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:04:43.911246 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.911242    5718 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:04:43.911379 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.911257    5718 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:04:43.912442 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.912425    5718 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:04:43.912442 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.912441    5718 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:04:43.913036 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.913016    5718 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:04:43.915969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.915954    5718 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:04:43.915969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.915967    5718 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:04:43.916074 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916066    5718 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 01:04:43.916266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916255    5718 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:04:43.916300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916285    5718 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:04:43.916385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916376    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:04:43.916413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916391    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:04:43.916413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916397    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:04:43.916413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916403    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:04:43.916413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916409    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916415    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916421    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916427    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916433    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916439    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916453    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916463    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916489    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:04:43.916517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.916495    5718 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:04:43.920019 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.920002    5718 server.go:1263] "Started kubelet"
Oct 31 01:04:43.920095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.920061    5718 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:04:43.920154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.920104    5718 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:04:43.920197 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.920182    5718 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:04:43.920573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.920542    5718 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:04:43.920675 ip-10-0-141-86 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:04:43.921416 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.921332    5718 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:04:43.925928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.925402    5718 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:04:43.925928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.925420    5718 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:04:43.925928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.925482    5718 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:33 +0000 UTC" deadline="2028-03-30 23:21:52.297374447 +0000 UTC"
Oct 31 01:04:43.925928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.925502    5718 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12406h17m8.371875082s"
Oct 31 01:04:43.927531 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.925553    5718 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:04:43.927531 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.926706    5718 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:04:43.927531 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:43.925555    5718 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:04:43.927531 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.925576    5718 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:04:43.929353 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929335    5718 factory.go:55] Registering systemd factory
Oct 31 01:04:43.929353 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929353    5718 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:04:43.929782 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929747    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120f0f3f-da9b-4395-807b-a291a37b5e2d" volumeName="kubernetes.io/secret/120f0f3f-da9b-4395-807b-a291a37b5e2d-monitoring-plugin-cert" seLinuxMountContext=""
Oct 31 01:04:43.929852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929787    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-client-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:43.929852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929802    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65337868-194c-4e3f-9e3f-7e96c35c6761" volumeName="kubernetes.io/projected/65337868-194c-4e3f-9e3f-7e96c35c6761-kube-api-access-6r9jn" seLinuxMountContext=""
Oct 31 01:04:43.929852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929813    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65337868-194c-4e3f-9e3f-7e96c35c6761" volumeName="kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 01:04:43.929852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929825    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961e1a63-66d7-4c5f-924e-27d9d56d6862" volumeName="kubernetes.io/projected/961e1a63-66d7-4c5f-924e-27d9d56d6862-kube-api-access-xpn68" seLinuxMountContext=""
Oct 31 01:04:43.929852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929838    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961e1a63-66d7-4c5f-924e-27d9d56d6862" volumeName="kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-tls" seLinuxMountContext=""
Oct 31 01:04:43.929852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929849    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929861    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929876    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05380ea7-efbf-4c24-9163-97a4243365df" volumeName="kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929888    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05380ea7-efbf-4c24-9163-97a4243365df" volumeName="kubernetes.io/projected/05380ea7-efbf-4c24-9163-97a4243365df-kube-api-access-x84vl" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929900    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a09fe47-166b-4fa0-9982-afffd73320b8" volumeName="kubernetes.io/projected/2a09fe47-166b-4fa0-9982-afffd73320b8-kube-api-access-2xjhl" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929913    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c464f6d-9424-478f-8b7c-c64b564d816f" volumeName="kubernetes.io/secret/2c464f6d-9424-478f-8b7c-c64b564d816f-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929925    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/projected/911363c3-235f-4667-b12c-94089715f1bc-kube-api-access-mmx7m" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929936    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929954    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961e1a63-66d7-4c5f-924e-27d9d56d6862" volumeName="kubernetes.io/empty-dir/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-textfile" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929965    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81" volumeName="kubernetes.io/projected/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-kube-api-access-2g25n" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929977    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65337868-194c-4e3f-9e3f-7e96c35c6761" volumeName="kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.929989    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e497701-d9b9-46b7-a5db-db294421d12f" volumeName="kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930001    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1608abd9-fd52-44e3-a945-eccf76f178aa" volumeName="kubernetes.io/secret/1608abd9-fd52-44e3-a945-eccf76f178aa-metrics-tls" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930013    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" volumeName="kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930026    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1608abd9-fd52-44e3-a945-eccf76f178aa" volumeName="kubernetes.io/projected/1608abd9-fd52-44e3-a945-eccf76f178aa-kube-api-access-29d76" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930039    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" volumeName="kubernetes.io/projected/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-kube-api-access-pc2r4" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930051    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930063    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961e1a63-66d7-4c5f-924e-27d9d56d6862" volumeName="kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930076    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" volumeName="kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930088    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930100    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa76e03-b876-4bc5-a0ec-aab3cced55f0" volumeName="kubernetes.io/secret/caa76e03-b876-4bc5-a0ec-aab3cced55f0-agent-certs" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930111    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930129    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930141    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930157    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs" seLinuxMountContext=""
Oct 31 01:04:43.930152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930171    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930183    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05380ea7-efbf-4c24-9163-97a4243365df" volumeName="kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930195    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603" volumeName="kubernetes.io/projected/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-kube-api-access-w2qs9" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930206    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb29d15c-7076-454a-a335-22e9b8097c4f" volumeName="kubernetes.io/projected/fb29d15c-7076-454a-a335-22e9b8097c4f-kube-api-access-f5lkz" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930218    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373" volumeName="kubernetes.io/projected/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-api-access-q9xcq" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930230    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373" volumeName="kubernetes.io/secret/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930243    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="792f66b0-e2d9-47c0-92bc-071ba29d52e5" volumeName="kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930256    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81" volumeName="kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930268    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930282    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930296    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930309    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373" volumeName="kubernetes.io/empty-dir/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-data-volume" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930330    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930342    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930354    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de7aadea-e0a6-4086-ae7a-a8f7794ad264" volumeName="kubernetes.io/projected/de7aadea-e0a6-4086-ae7a-a8f7794ad264-kube-api-access-gn884" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930366    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9562f13-b930-4947-9aac-9379535e8fe6" volumeName="kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-multus-daemon-config" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930378    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930391    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930404    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930418    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3cee94-19dc-4e56-b7a2-256765111acb" volumeName="kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-tmp" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930430    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/empty-dir/911363c3-235f-4667-b12c-94089715f1bc-tmp" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930446    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051a847-0102-43c1-8fd4-7885e50459c2" volumeName="kubernetes.io/projected/f051a847-0102-43c1-8fd4-7885e50459c2-kube-api-access-kbtpv" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930460    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930474    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3cee94-19dc-4e56-b7a2-256765111acb" volumeName="kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-tuned" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930488    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9562f13-b930-4947-9aac-9379535e8fe6" volumeName="kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930513    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9562f13-b930-4947-9aac-9379535e8fe6" volumeName="kubernetes.io/projected/e9562f13-b930-4947-9aac-9379535e8fe6-kube-api-access-p46tq" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930526    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3cee94-19dc-4e56-b7a2-256765111acb" volumeName="kubernetes.io/projected/7c3cee94-19dc-4e56-b7a2-256765111acb-kube-api-access-769kz" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930539    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/configmap/911363c3-235f-4667-b12c-94089715f1bc-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930554    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930568    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930580    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930602    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1608abd9-fd52-44e3-a945-eccf76f178aa" volumeName="kubernetes.io/empty-dir/1608abd9-fd52-44e3-a945-eccf76f178aa-tmp-dir" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930618    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a09fe47-166b-4fa0-9982-afffd73320b8" volumeName="kubernetes.io/configmap/2a09fe47-166b-4fa0-9982-afffd73320b8-iptables-alerter-script" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930630    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961e1a63-66d7-4c5f-924e-27d9d56d6862" volumeName="kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930643    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930687    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930700    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c464f6d-9424-478f-8b7c-c64b564d816f" volumeName="kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-config" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930711    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930723    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930734    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930747    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de7aadea-e0a6-4086-ae7a-a8f7794ad264" volumeName="kubernetes.io/empty-dir/de7aadea-e0a6-4086-ae7a-a8f7794ad264-tmp-dir" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930759    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05380ea7-efbf-4c24-9163-97a4243365df" volumeName="kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930771    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120f0f3f-da9b-4395-807b-a291a37b5e2d" volumeName="kubernetes.io/projected/120f0f3f-da9b-4395-807b-a291a37b5e2d-kube-api-access-bkf97" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930782    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1608abd9-fd52-44e3-a945-eccf76f178aa" volumeName="kubernetes.io/configmap/1608abd9-fd52-44e3-a945-eccf76f178aa-config-volume" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930793    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c464f6d-9424-478f-8b7c-c64b564d816f" volumeName="kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-env-overrides" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930805    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c464f6d-9424-478f-8b7c-c64b564d816f" volumeName="kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930817    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930830    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-grpc-tls" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930842    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930854    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05380ea7-efbf-4c24-9163-97a4243365df" volumeName="kubernetes.io/empty-dir/05380ea7-efbf-4c24-9163-97a4243365df-volume-directive-shadow" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930866    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373" volumeName="kubernetes.io/configmap/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930879    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65337868-194c-4e3f-9e3f-7e96c35c6761" volumeName="kubernetes.io/configmap/65337868-194c-4e3f-9e3f-7e96c35c6761-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930891    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa76e03-b876-4bc5-a0ec-aab3cced55f0" volumeName="kubernetes.io/configmap/caa76e03-b876-4bc5-a0ec-aab3cced55f0-konnectivity-ca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930904    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930916    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930928    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c464f6d-9424-478f-8b7c-c64b564d816f" volumeName="kubernetes.io/projected/2c464f6d-9424-478f-8b7c-c64b564d816f-kube-api-access-jxmqv" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930941    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" volumeName="kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930953    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05380ea7-efbf-4c24-9163-97a4243365df" volumeName="kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930965    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930978    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961e1a63-66d7-4c5f-924e-27d9d56d6862" volumeName="kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.930990    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603" volumeName="kubernetes.io/configmap/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-serviceca" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931002    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/empty-dir/432f223d-4785-4a5c-af6a-921a65b1a08e-audit-log" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931014    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="432f223d-4785-4a5c-af6a-921a65b1a08e" volumeName="kubernetes.io/projected/432f223d-4785-4a5c-af6a-921a65b1a08e-kube-api-access-gz8n7" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931027    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" volumeName="kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931048    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="911363c3-235f-4667-b12c-94089715f1bc" volumeName="kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931063    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931076    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbfe986f-2a26-4c74-9564-dbb4585d53cd" volumeName="kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931088    5718 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051a847-0102-43c1-8fd4-7885e50459c2" volumeName="kubernetes.io/secret/f051a847-0102-43c1-8fd4-7885e50459c2-cert" seLinuxMountContext=""
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931099    5718 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:04:43.931411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.931107    5718 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:04:43.934365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.932038    5718 factory.go:153] Registering CRI-O factory
Oct 31 01:04:43.934365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.932059    5718 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:04:43.934365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.932123    5718 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:04:43.934365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.932144    5718 factory.go:103] Registering Raw factory
Oct 31 01:04:43.934365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.932159    5718 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:04:43.934365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.932525    5718 manager.go:319] Starting recovery of all containers
Oct 31 01:04:43.957998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.957946    5718 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:04:43.959507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.959483    5718 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:04:43.959507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.959512    5718 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:04:43.959782 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.959535    5718 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:04:43.959782 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:43.959571    5718 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 01:04:43.970418 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.970386    5718 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="a0b4ce9a252479a912627f8c64fb4e4ee27db4cb0e2b24e60adbd6c5c226885a" exitCode=0
Oct 31 01:04:43.970418 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.970417    5718 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="f3f513c01cb242c79ba14e31e9161d24c2c09dbad86e38369f929d6fdf740b83" exitCode=0
Oct 31 01:04:43.970595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.970426    5718 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="38c828ed6ea35a73bad82cbd882e226d576f366a314271c7c56b23051b95b88e" exitCode=0
Oct 31 01:04:43.970595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.970434    5718 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="8309b6b79befd33427b8496efa62679666407b972566891d3077d4352c3fd3bc" exitCode=0
Oct 31 01:04:43.970595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.970442    5718 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="59cb839dd41293b00297f334d6fcbf5cc4dda349688f1eaf88de46f2dbbcf95f" exitCode=0
Oct 31 01:04:43.970595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.970450    5718 generic.go:356] "Generic (PLEG): container finished" podID="1e8405b7-4f76-495c-b9fc-04affdf8d1f8" containerID="b89359fb9997023ab2cae6f9871d845164f36d61bb782f660d851a580ee2484e" exitCode=0
Oct 31 01:04:43.984126 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:43.984099    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="59b46adfead674c24e08869acb4255dd10bb6fdf2c26c6a0462c018cc49d1f80" exitCode=0
Oct 31 01:04:44.010071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.010029    5718 generic.go:356] "Generic (PLEG): container finished" podID="8ed86943c8c1fe4dccb7ac2c7a181775" containerID="e621b79a12a4ee8ca0b6947930ae18faed019554e8c949ee7df3a436cfd6291b" exitCode=0
Oct 31 01:04:44.017492 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.017460    5718 generic.go:356] "Generic (PLEG): container finished" podID="961e1a63-66d7-4c5f-924e-27d9d56d6862" containerID="1e0f173caccdc0c3e21a41b406c6baa72384c678b9e05f5e81036b9c2330d938" exitCode=0
Oct 31 01:04:44.026904 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.026840    5718 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:04:44.050382 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.050359    5718 manager.go:324] Recovery completed
Oct 31 01:04:44.059691 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.059665    5718 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 01:04:44.071508 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.071489    5718 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:04:44.073445 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.073422    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:04:44.073542 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.073464    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:04:44.073542 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.073480    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:04:44.078307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078276    5718 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:04:44.078307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078293    5718 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:04:44.078307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078278    5718 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:04:44.078307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078312    5718 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:04:44.078810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078793    5718 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 01:04:44.078854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078810    5718 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 01:04:44.078854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078828    5718 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 01:04:44.078854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078834    5718 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 01:04:44.078854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078840    5718 policy_none.go:50] "Start"
Oct 31 01:04:44.078854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078850    5718 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:04:44.079044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078861    5718 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:04:44.079044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.078994    5718 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 01:04:44.079044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.079001    5718 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 01:04:44.079044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.079007    5718 policy_none.go:44] "Start"
Oct 31 01:04:44.079642 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.079618    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:04:44.079704 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.079674    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:04:44.079704 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.079686    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:04:44.084047 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.084017    5718 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:04:44.086762 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.086750    5718 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:04:44.086812 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.086777    5718 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:04:44.086812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.086795    5718 server.go:87] "Starting device plugin registration server"
Oct 31 01:04:44.087151 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087140    5718 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:04:44.087192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087154    5718 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:04:44.087279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087257    5718 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:04:44.087399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087371    5718 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:04:44.087447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087415    5718 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:04:44.087447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087422    5718 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:04:44.088360 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.087478    5718 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:04:44.087397036Z","UUID":"e1fdabe2-ff1c-47ae-8a68-40c6a34caff7","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:04:44.089498 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.089307    5718 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:04:44.089498 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.089351    5718 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-86.us-west-1.compute.internal\" not found"
Oct 31 01:04:44.089683 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.089566    5718 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:04:44.089683 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.089590    5718 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:04:44.127834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.127800    5718 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:44.127834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.127814    5718 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:44.130941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.130921    5718 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 01:04:44.171643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.171613    5718 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:44.187881 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.187852    5718 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:04:44.188983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.188963    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:04:44.189069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.188995    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:04:44.189069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.189007    5718 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:04:44.189069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.189060    5718 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.229019 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.228986    5718 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:44.260192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260119    5718 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"]
Oct 31 01:04:44.260845 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260809    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"fafe7c61d6af085c60e65c5248ab0754","Type":"ContainerStarted","Data":"be0a86f8f7f2590fc573bd7cb3ecff27f171b442ce5498951a448cb61955a74c"}
Oct 31 01:04:44.260906 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260851    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"fafe7c61d6af085c60e65c5248ab0754","Type":"ContainerStarted","Data":"234df9043fcded13103b146179cf2e43c4fd6107ca3815f637b497b95455553e"}
Oct 31 01:04:44.260906 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260877    5718 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.261039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260933    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"8ed86943c8c1fe4dccb7ac2c7a181775","Type":"ContainerStarted","Data":"f5f3c108c75a0d2e9490fabda37d5d0611692bf39c79a522f8b57b44b280556e"}
Oct 31 01:04:44.261039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260947    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"8ed86943c8c1fe4dccb7ac2c7a181775","Type":"ContainerDied","Data":"e621b79a12a4ee8ca0b6947930ae18faed019554e8c949ee7df3a436cfd6291b"}
Oct 31 01:04:44.261039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.260962    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal" event={"ID":"8ed86943c8c1fe4dccb7ac2c7a181775","Type":"ContainerStarted","Data":"53ac33fd4237cc94cbe957e60f438f31bd77d411ae75b19f2a60d0e88bbe43ce"}
Oct 31 01:04:44.261130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.261067    5718 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.299687 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.299600    5718 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.299819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.299708    5718 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.322313 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.322281    5718 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:44.322452 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.322346    5718 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.322542 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.322528    5718 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:44.322572 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:44.322560    5718 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.332712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.332644    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.332855 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.332728    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.332855 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.332758    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fafe7c61d6af085c60e65c5248ab0754-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"fafe7c61d6af085c60e65c5248ab0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.433745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.433708    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.433745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.433744    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.433954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.433771    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fafe7c61d6af085c60e65c5248ab0754-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"fafe7c61d6af085c60e65c5248ab0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.433954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.433815    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fafe7c61d6af085c60e65c5248ab0754-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"fafe7c61d6af085c60e65c5248ab0754\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.433954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.433836    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.433954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.433815    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8ed86943c8c1fe4dccb7ac2c7a181775-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" (UID: \"8ed86943c8c1fe4dccb7ac2c7a181775\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:44.911974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:44.911933    5718 apiserver.go:52] "Watching apiserver"
Oct 31 01:04:45.027332 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.027308    5718 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:45.068665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.068626    5718 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:04:45.069371 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.069348    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-psw7z","openshift-ingress-canary/ingress-canary-ntdx6","openshift-image-registry/node-ca-vkb7h","openshift-network-operator/iptables-alerter-rrnxg","openshift-ovn-kubernetes/ovnkube-node-rm9cp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg","openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5","openshift-insights/insights-runtime-extractor-mr6h9","openshift-monitoring/monitoring-plugin-77f4f76594-cc59x","openshift-monitoring/node-exporter-tm9rb","openshift-multus/multus-additional-cni-plugins-9lpwg","openshift-network-diagnostics/network-check-target-fczdh","kube-system/konnectivity-agent-j6h6v","openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp","openshift-multus/multus-68tmm","kube-system/global-pull-secret-syncer-pf5kd","openshift-console/console-6b5d58f5bd-vchll","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-rdsvq","openshift-dns/node-resolver-jjdkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal","openshift-monitoring/metrics-server-66f6f89fb8-2wbfh","openshift-dns/dns-default-hmdlr","openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"]
Oct 31 01:04:45.086867 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.086837    5718 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:45.087036 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:04:45.086888    5718 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-1.compute.internal"
Oct 31 01:04:45.123555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123526    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:04:45.123876 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123845    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123860    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123888    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123866    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123906    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123908    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123883    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.123983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.123982    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:04:45.124282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124007    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.124282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124126    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.124356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124303    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:04:45.124540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124527    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9z86\""
Oct 31 01:04:45.124622 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124551    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:04:45.124696 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124668    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-55dd6\""
Oct 31 01:04:45.124747 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124696    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kk499\""
Oct 31 01:04:45.124839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.124823    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:04:45.125257 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125231    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:04:45.125365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125347    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42fft\""
Oct 31 01:04:45.125434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125366    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.125483 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125466    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:04:45.125695 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125679    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:04:45.125778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125766    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.125822 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125796    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.125822 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125801    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rnpg9\""
Oct 31 01:04:45.125941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125906    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g746\""
Oct 31 01:04:45.125973 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.125943    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.126069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126049    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8rwd\""
Oct 31 01:04:45.126115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126067    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.126115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126080    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.126183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126128    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.126183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126140    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:04:45.126245 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126186    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:04:45.126336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126312    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.126493 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126402    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:04:45.126640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126615    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:04:45.126935 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.126917    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:04:45.127279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127257    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zzhf8\""
Oct 31 01:04:45.127279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127267    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-grw5x\""
Oct 31 01:04:45.127406 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127301    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:04:45.127406 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127312    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:04:45.127406 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127338    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:04:45.127406 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127342    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:04:45.127888 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.127868    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sjdn\""
Oct 31 01:04:45.128223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.128209    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.128223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.128213    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xgzbp\""
Oct 31 01:04:45.128331 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.128273    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:04:45.128362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.128349    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.128393 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.128365    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:04:45.129205 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.129193    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.129448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.129434    5718 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:04:45.129918 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.129901    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:04:45.130021 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.130008    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:04:45.130210 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.130200    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.130824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.130806    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:04:45.130868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.130814    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvb7t\""
Oct 31 01:04:45.130919 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.130902    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.130956 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.130919    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 01:04:45.137346 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137325    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-metrics-client-ca\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.137455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137360    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911363c3-235f-4667-b12c-94089715f1bc-metrics-client-ca\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.137455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137389    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.137540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137493    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-netd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.137540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137522    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:45.137605 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137556    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-serviceca\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.137605 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137591    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a09fe47-166b-4fa0-9982-afffd73320b8-iptables-alerter-script\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.137733 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137612    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.137787 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137739    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-crio-socket\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.137833 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137787    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-root\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.137875 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137835    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.137875 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137842    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a09fe47-166b-4fa0-9982-afffd73320b8-iptables-alerter-script\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.137962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137888    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.137962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137897    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-metrics-certs\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:45.138070 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.137969    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.138070 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138056    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.138161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138073    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a09fe47-166b-4fa0-9982-afffd73320b8-host-slash\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.138161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138092    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lkz\" (UniqueName: \"kubernetes.io/projected/fb29d15c-7076-454a-a335-22e9b8097c4f-kube-api-access-f5lkz\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.138161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138108    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.138161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138125    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:45.138161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138154    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138180    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-kubelet\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138178    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138207    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138243    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138272    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46tq\" (UniqueName: \"kubernetes.io/projected/e9562f13-b930-4947-9aac-9379535e8fe6-kube-api-access-p46tq\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138297    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-tmp\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138345    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7aadea-e0a6-4086-ae7a-a8f7794ad264-tmp-dir\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138344    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-serviceca\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138364    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjhl\" (UniqueName: \"kubernetes.io/projected/2a09fe47-166b-4fa0-9982-afffd73320b8-kube-api-access-2xjhl\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.138385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138385    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysconfig\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138402    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138409    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-system-cni-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138434    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138439    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-tmp\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138450    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7aadea-e0a6-4086-ae7a-a8f7794ad264-tmp-dir\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138509    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-cni-binary-copy\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138554    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-log-socket\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138617    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g25n\" (UniqueName: \"kubernetes.io/projected/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-kube-api-access-2g25n\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138673    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-cni-binary-copy\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138679    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-conf-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138708    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911363c3-235f-4667-b12c-94089715f1bc-tmp\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138738    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138765    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qs9\" (UniqueName: \"kubernetes.io/projected/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-kube-api-access-w2qs9\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138790    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-multus-daemon-config\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138820    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911363c3-235f-4667-b12c-94089715f1bc-tmp\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.138823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138830    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-systemd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138859    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-env-overrides\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138908    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-kubelet-config\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138945    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-host\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.138978    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-sys-fs\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139026    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-bin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139032    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9562f13-b930-4947-9aac-9379535e8fe6-multus-daemon-config\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139027    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-env-overrides\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139077    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-sys\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139097    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139118    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9jn\" (UniqueName: \"kubernetes.io/projected/65337868-194c-4e3f-9e3f-7e96c35c6761-kube-api-access-6r9jn\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139210    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de7aadea-e0a6-4086-ae7a-a8f7794ad264-hosts-file\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139229    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-cnibin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139247    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-hostroot\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139268    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xcq\" (UniqueName: \"kubernetes.io/projected/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-api-access-q9xcq\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139295    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139315    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139367    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-run\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139399    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-client-certs\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139433    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139495    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.139495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139494    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e497701-d9b9-46b7-a5db-db294421d12f-original-pull-secret\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139520    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cni-binary-copy\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139562    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-accelerators-collector-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139606    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1608abd9-fd52-44e3-a945-eccf76f178aa-metrics-tls\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139701    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-etc-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139741    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139778    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c464f6d-9424-478f-8b7c-c64b564d816f-ovn-node-metrics-cert\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139867    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-metrics-server-audit-profiles\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139900    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139927    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139940    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c464f6d-9424-478f-8b7c-c64b564d816f-ovn-node-metrics-cert\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139965    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139990    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.139995    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-socket-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140023    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-multus-certs\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140054    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140112    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140159    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-lib-modules\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140182    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140228    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-data-volume\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140244    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-tls\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140260    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05380ea7-efbf-4c24-9163-97a4243365df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140278    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-conf\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.140334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140293    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1608abd9-fd52-44e3-a945-eccf76f178aa-tmp-dir\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140356    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1608abd9-fd52-44e3-a945-eccf76f178aa-tmp-dir\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140366    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-data-volume\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140370    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05380ea7-efbf-4c24-9163-97a4243365df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140399    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-systemd-units\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140428    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-slash\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140447    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140465    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140503    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf97\" (UniqueName: \"kubernetes.io/projected/120f0f3f-da9b-4395-807b-a291a37b5e2d-kube-api-access-bkf97\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140563    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-client-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140601    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140634    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140683    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140706    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-tuned\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140737    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140763    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-socket-dir-parent\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140771    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-tuned\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140785    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140829    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140852    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-var-lib-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140872    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/120f0f3f-da9b-4395-807b-a291a37b5e2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140888    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/432f223d-4785-4a5c-af6a-921a65b1a08e-audit-log\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140924    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-ovn\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140949    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.140982    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141005    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/432f223d-4785-4a5c-af6a-921a65b1a08e-audit-log\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141017    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141073    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141099    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141123    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-769kz\" (UniqueName: \"kubernetes.io/projected/7c3cee94-19dc-4e56-b7a2-256765111acb-kube-api-access-769kz\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141142    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa76e03-b876-4bc5-a0ec-aab3cced55f0-konnectivity-ca\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141170    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn884\" (UniqueName: \"kubernetes.io/projected/de7aadea-e0a6-4086-ae7a-a8f7794ad264-kube-api-access-gn884\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141201    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141226    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-netns\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.141249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141251    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-textfile\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141317    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa76e03-b876-4bc5-a0ec-aab3cced55f0-konnectivity-ca\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141321    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1608abd9-fd52-44e3-a945-eccf76f178aa-config-volume\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141346    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-textfile\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141361    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-node-log\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141390    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141425    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-modprobe-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141447    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cnibin\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141465    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa76e03-b876-4bc5-a0ec-aab3cced55f0-agent-certs\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141472    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1608abd9-fd52-44e3-a945-eccf76f178aa-config-volume\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141487    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtpv\" (UniqueName: \"kubernetes.io/projected/f051a847-0102-43c1-8fd4-7885e50459c2-kube-api-access-kbtpv\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141541    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65337868-194c-4e3f-9e3f-7e96c35c6761-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141588    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141644    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa76e03-b876-4bc5-a0ec-aab3cced55f0-agent-certs\") pod \"konnectivity-agent-j6h6v\" (UID: \"caa76e03-b876-4bc5-a0ec-aab3cced55f0\") " pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141692    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-system-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141711    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-kubelet\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141730    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-script-lib\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141751    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-dbus\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141782    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141836    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-multus\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141897    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141936    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-systemd\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141970    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29d76\" (UniqueName: \"kubernetes.io/projected/1608abd9-fd52-44e3-a945-eccf76f178aa-kube-api-access-29d76\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141992    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-script-lib\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.141996    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-bin\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142035    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-host\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142097    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142114    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f051a847-0102-43c1-8fd4-7885e50459c2-cert\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142146    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84vl\" (UniqueName: \"kubernetes.io/projected/05380ea7-efbf-4c24-9163-97a4243365df-kube-api-access-x84vl\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142195    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2r4\" (UniqueName: \"kubernetes.io/projected/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-kube-api-access-pc2r4\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.142269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142293    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142301    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f051a847-0102-43c1-8fd4-7885e50459c2-cert\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142333    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-sys\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142359    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142380    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142400    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142438    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142434    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-registration-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142471    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpn68\" (UniqueName: \"kubernetes.io/projected/961e1a63-66d7-4c5f-924e-27d9d56d6862-kube-api-access-xpn68\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142471    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65337868-194c-4e3f-9e3f-7e96c35c6761-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142505    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142616    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142687    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142724    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142751    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142773    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-netns\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142790    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmqv\" (UniqueName: \"kubernetes.io/projected/2c464f6d-9424-478f-8b7c-c64b564d816f-kube-api-access-jxmqv\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142854    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-tls\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142874    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8n7\" (UniqueName: \"kubernetes.io/projected/432f223d-4785-4a5c-af6a-921a65b1a08e-kube-api-access-gz8n7\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142891    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142917    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-k8s-cni-cncf-io\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142943    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-etc-kubernetes\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.142996    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-wtmp\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143024    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-grpc-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143061    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-config\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143085    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143105    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143126    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143144    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-var-lib-kubelet\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143188    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.143222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143224    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-device-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.144087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143251    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmx7m\" (UniqueName: \"kubernetes.io/projected/911363c3-235f-4667-b12c-94089715f1bc-kube-api-access-mmx7m\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.144087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143267    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-kubernetes\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.144087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143276    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.144087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.143276    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c464f6d-9424-478f-8b7c-c64b564d816f-ovnkube-config\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.173892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.173811    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.174042 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174011    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:04:45.174341 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174323    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 01:04:45.174574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174559    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 01:04:45.174671 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174570    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:04:45.174671 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174597    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:04:45.174776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174646    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7hnm9\""
Oct 31 01:04:45.174776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174675    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:04:45.174776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174709    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8c8qv8uqffkk1\""
Oct 31 01:04:45.174776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174725    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xs8jk\""
Oct 31 01:04:45.174776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174764    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:04:45.174925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174858    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 01:04:45.174925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174907    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 01:04:45.174925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174909    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 01:04:45.175037 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174910    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ln2ts\""
Oct 31 01:04:45.175037 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.174952    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 01:04:45.175180 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175164    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 01:04:45.175236 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175201    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 01:04:45.175293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175243    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 01:04:45.175343 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175324    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 01:04:45.175390 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175350    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 01:04:45.175390 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175352    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 01:04:45.175495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.175481    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 01:04:45.177837 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.177815    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911363c3-235f-4667-b12c-94089715f1bc-metrics-client-ca\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.177951 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.177888    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-metrics-client-ca\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.178113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.178098    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 01:04:45.178946 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.178927    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 01:04:45.179123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.179107    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180684    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-tls\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180699    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180690    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180730    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-accelerators-collector-config\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180747    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180748    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180751    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-metrics-server-audit-profiles\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.180809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180783    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1608abd9-fd52-44e3-a945-eccf76f178aa-metrics-tls\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.181111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180878    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.181111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.180878    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.181549 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.181532    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.181745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.181727    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05380ea7-efbf-4c24-9163-97a4243365df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.181783 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.181759    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.181831 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.181815    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.181864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.181851    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/432f223d-4785-4a5c-af6a-921a65b1a08e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.181899 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.181883    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65337868-194c-4e3f-9e3f-7e96c35c6761-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.183134 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.183117    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.183134 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.183127    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05380ea7-efbf-4c24-9163-97a4243365df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.183248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.183165    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.183289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.183282    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-grpc-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.183372 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.183358    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.195059 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.195033    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lkz\" (UniqueName: \"kubernetes.io/projected/fb29d15c-7076-454a-a335-22e9b8097c4f-kube-api-access-f5lkz\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.195185 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.195169    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmslr\" (UniqueName: \"kubernetes.io/projected/792f66b0-e2d9-47c0-92bc-071ba29d52e5-kube-api-access-jmslr\") pod \"network-check-target-fczdh\" (UID: \"792f66b0-e2d9-47c0-92bc-071ba29d52e5\") " pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:45.196166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.196148    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g25n\" (UniqueName: \"kubernetes.io/projected/bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81-kube-api-access-2g25n\") pod \"network-metrics-daemon-rdsvq\" (UID: \"bf64e4cb-1bd2-4a65-8d05-fb0d85be1f81\") " pod="openshift-multus/network-metrics-daemon-rdsvq"
Oct 31 01:04:45.197347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.197118    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xcq\" (UniqueName: \"kubernetes.io/projected/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-kube-api-access-q9xcq\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.197347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.197166    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qs9\" (UniqueName: \"kubernetes.io/projected/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-kube-api-access-w2qs9\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.197347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.197227    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjhl\" (UniqueName: \"kubernetes.io/projected/2a09fe47-166b-4fa0-9982-afffd73320b8-kube-api-access-2xjhl\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.197910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.197605    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9jn\" (UniqueName: \"kubernetes.io/projected/65337868-194c-4e3f-9e3f-7e96c35c6761-kube-api-access-6r9jn\") pod \"openshift-state-metrics-7bd7489957-h5ltp\" (UID: \"65337868-194c-4e3f-9e3f-7e96c35c6761\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-h5ltp"
Oct 31 01:04:45.197910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.197828    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf97\" (UniqueName: \"kubernetes.io/projected/120f0f3f-da9b-4395-807b-a291a37b5e2d-kube-api-access-bkf97\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:45.198054 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.197974    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.198054 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.198030    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46tq\" (UniqueName: \"kubernetes.io/projected/e9562f13-b930-4947-9aac-9379535e8fe6-kube-api-access-p46tq\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.199068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.199047    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.206865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.202748    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn884\" (UniqueName: \"kubernetes.io/projected/de7aadea-e0a6-4086-ae7a-a8f7794ad264-kube-api-access-gn884\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.206865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.202780    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-769kz\" (UniqueName: \"kubernetes.io/projected/7c3cee94-19dc-4e56-b7a2-256765111acb-kube-api-access-769kz\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.206865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.202883    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtpv\" (UniqueName: \"kubernetes.io/projected/f051a847-0102-43c1-8fd4-7885e50459c2-kube-api-access-kbtpv\") pod \"ingress-canary-ntdx6\" (UID: \"f051a847-0102-43c1-8fd4-7885e50459c2\") " pod="openshift-ingress-canary/ingress-canary-ntdx6"
Oct 31 01:04:45.207700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.207590    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmqv\" (UniqueName: \"kubernetes.io/projected/2c464f6d-9424-478f-8b7c-c64b564d816f-kube-api-access-jxmqv\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.207700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.207622    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpn68\" (UniqueName: \"kubernetes.io/projected/961e1a63-66d7-4c5f-924e-27d9d56d6862-kube-api-access-xpn68\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.207824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.207768    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29d76\" (UniqueName: \"kubernetes.io/projected/1608abd9-fd52-44e3-a945-eccf76f178aa-kube-api-access-29d76\") pod \"dns-default-hmdlr\" (UID: \"1608abd9-fd52-44e3-a945-eccf76f178aa\") " pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:45.207857 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.207840    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84vl\" (UniqueName: \"kubernetes.io/projected/05380ea7-efbf-4c24-9163-97a4243365df-kube-api-access-x84vl\") pod \"kube-state-metrics-5dc8d7d89b-rk4z5\" (UID: \"05380ea7-efbf-4c24-9163-97a4243365df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-rk4z5"
Oct 31 01:04:45.208352 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.208329    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmx7m\" (UniqueName: \"kubernetes.io/projected/911363c3-235f-4667-b12c-94089715f1bc-kube-api-access-mmx7m\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.208414 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.208377    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8n7\" (UniqueName: \"kubernetes.io/projected/432f223d-4785-4a5c-af6a-921a65b1a08e-kube-api-access-gz8n7\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.208776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.208751    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2r4\" (UniqueName: \"kubernetes.io/projected/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-kube-api-access-pc2r4\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.225954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.225926    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 01:04:45.226736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.226716    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ch46h\""
Oct 31 01:04:45.226736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.226721    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 01:04:45.227037 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227021    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 01:04:45.227111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227021    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 01:04:45.227111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227082    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 01:04:45.227111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227090    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 01:04:45.227111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227090    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227109    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227127    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227126    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227164    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227183    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227251    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wdn2p\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227292    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsmab2l1uuhg0\""
Oct 31 01:04:45.227295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227298    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227377    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zmtbq\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227388    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227416    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vk6v8\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227437    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227458    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227550    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227607    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mdbs5\""
Oct 31 01:04:45.227705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227707    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 01:04:45.227972 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227724    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-36k3cvsjqf5vk\""
Oct 31 01:04:45.227972 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.227789    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 01:04:45.228820 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.228801    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.228940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.228805    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.228940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.228836    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.228940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.228836    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.229105 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.228958    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.230115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.230095    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-client-certs\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.230115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.230106    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.230253 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.230153    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") pod \"console-6b5d58f5bd-vchll\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") " pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:45.231289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.231265    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/120f0f3f-da9b-4395-807b-a291a37b5e2d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cc59x\" (UID: \"120f0f3f-da9b-4395-807b-a291a37b5e2d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:45.231387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.231359    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.231444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.231399    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-client-ca-bundle\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.231444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.231418    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.231444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.231398    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-tls\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.232561 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.232542    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.232683 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.232564    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.232683 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.232678    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.232810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.232796    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.233742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.233721    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.233897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.233749    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/911363c3-235f-4667-b12c-94089715f1bc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f6f54b69d-l5mz6\" (UID: \"911363c3-235f-4667-b12c-94089715f1bc\") " pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:45.233897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.233790    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/432f223d-4785-4a5c-af6a-921a65b1a08e-secret-metrics-server-tls\") pod \"metrics-server-66f6f89fb8-2wbfh\" (UID: \"432f223d-4785-4a5c-af6a-921a65b1a08e\") " pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:04:45.233897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.233815    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:45.243716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243690    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-kubernetes\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.243834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243723    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.243834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243744    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-netd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.243834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243804    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-netd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.243834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243804    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-kubernetes\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.243834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243807    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.243834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243826    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243855    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-crio-socket\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243879    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-root\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243917    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-root\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243935    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373-crio-socket\") pod \"insights-runtime-extractor-mr6h9\" (UID: \"5adc1e9b-39f1-4a9e-bbc6-1a5f8fdf6373\") " pod="openshift-insights/insights-runtime-extractor-mr6h9"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243952    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.243987    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a09fe47-166b-4fa0-9982-afffd73320b8-host-slash\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244022    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a09fe47-166b-4fa0-9982-afffd73320b8-host-slash\") pod \"iptables-alerter-rrnxg\" (UID: \"2a09fe47-166b-4fa0-9982-afffd73320b8\") " pod="openshift-network-operator/iptables-alerter-rrnxg"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244025    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-kubelet\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244050    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244074    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244104    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysconfig\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244118    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-kubelet\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244127    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-system-cni-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.244140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244134    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244152    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-log-socket\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244159    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244163    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysconfig\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244175    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-system-cni-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244176    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-conf-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244201    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-log-socket\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244209    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-systemd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244212    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-conf-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244232    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-kubelet-config\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244256    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-host\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244258    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-systemd\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244280    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-sys-fs\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244299    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-bin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244299    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-kubelet-config\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244319    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-sys\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244340    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-sys-fs\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244348    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-sys\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244300    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603-host\") pod \"node-ca-vkb7h\" (UID: \"c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603\") " pod="openshift-image-registry/node-ca-vkb7h"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244353    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de7aadea-e0a6-4086-ae7a-a8f7794ad264-hosts-file\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244376    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-bin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244383    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de7aadea-e0a6-4086-ae7a-a8f7794ad264-hosts-file\") pod \"node-resolver-jjdkz\" (UID: \"de7aadea-e0a6-4086-ae7a-a8f7794ad264\") " pod="openshift-dns/node-resolver-jjdkz"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244385    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-cnibin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244418    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-cnibin\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244446    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-hostroot\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244473    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-hostroot\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244502    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-run\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244521    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-etc-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244539    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244569    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-socket-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244577    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-run\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244598    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244572    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-etc-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244601    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-multus-certs\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244633    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-socket-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244637    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-multus-certs\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244666    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-lib-modules\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244698    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-conf\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244729    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-systemd-units\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244749    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-slash\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244772    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244782    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-lib-modules\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244801    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-socket-dir-parent\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244806    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-systemd-units\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244822    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-slash\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244825    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-var-lib-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.244812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244853    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-multus-socket-dir-parent\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244864    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-ovn\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244880    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-var-lib-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244885    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-netns\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244905    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244912    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-node-log\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244914    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-sysctl-conf\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244912    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-ovn\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244934    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-netns\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244967    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-modprobe-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.244999    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-node-log\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245008    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cnibin\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245051    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-modprobe-d\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245052    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-cnibin\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245084    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-system-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245099    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-kubelet\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245114    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-dbus\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245118    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-system-cni-dir\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245131    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-multus\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245151    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-systemd\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245165    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-kubelet\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245168    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-bin\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245190    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-cni-bin\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245175    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e497701-d9b9-46b7-a5db-db294421d12f-dbus\") pod \"global-pull-secret-syncer-pf5kd\" (UID: \"2e497701-d9b9-46b7-a5db-db294421d12f\") " pod="kube-system/global-pull-secret-syncer-pf5kd"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245201    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-host\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245208    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-var-lib-cni-multus\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245230    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-sys\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245235    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-etc-systemd\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245243    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-host\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245250    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-registration-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245260    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-sys\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245267    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245276    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-registration-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245288    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245315    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-netns\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245324    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e8405b7-4f76-495c-b9fc-04affdf8d1f8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9lpwg\" (UID: \"1e8405b7-4f76-495c-b9fc-04affdf8d1f8\") " pod="openshift-multus/multus-additional-cni-plugins-9lpwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245345    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-k8s-cni-cncf-io\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245360    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-run-openvswitch\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245375    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-etc-kubernetes\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245363    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c464f6d-9424-478f-8b7c-c64b564d816f-host-run-netns\") pod \"ovnkube-node-rm9cp\" (UID: \"2c464f6d-9424-478f-8b7c-c64b564d816f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245398    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-wtmp\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245404    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-host-run-k8s-cni-cncf-io\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245417    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-var-lib-kubelet\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245423    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9562f13-b930-4947-9aac-9379535e8fe6-etc-kubernetes\") pod \"multus-68tmm\" (UID: \"e9562f13-b930-4947-9aac-9379535e8fe6\") " pod="openshift-multus/multus-68tmm"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245435    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-device-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245442    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961e1a63-66d7-4c5f-924e-27d9d56d6862-node-exporter-wtmp\") pod \"node-exporter-tm9rb\" (UID: \"961e1a63-66d7-4c5f-924e-27d9d56d6862\") " pod="openshift-monitoring/node-exporter-tm9rb"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245457    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3cee94-19dc-4e56-b7a2-256765111acb-var-lib-kubelet\") pod \"tuned-psw7z\" (UID: \"7c3cee94-19dc-4e56-b7a2-256765111acb\") " pod="openshift-cluster-node-tuning-operator/tuned-psw7z"
Oct 31 01:04:45.246959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.245497    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb29d15c-7076-454a-a335-22e9b8097c4f-device-dir\") pod \"aws-ebs-csi-driver-node-ltvwg\" (UID: \"fb29d15c-7076-454a-a335-22e9b8097c4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ltvwg"
Oct 31 01:04:45.606925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:45.606894    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cc59x"
Oct 31 01:04:46.255064 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:46.255027    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:04:47.965811 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:47.965775    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmdlr"
Oct 31 01:04:48.040221 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:48.040185    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:48.066031 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:48.065997    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:04:50.209802 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:50.209769    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f6f54b69d-l5mz6"
Oct 31 01:04:51.527374 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:51.527342    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:04:52.762130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:52.762061    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:53.140012 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:53.139932    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5d58f5bd-vchll"]
Oct 31 01:04:54.222154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:54.222120    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:04:54.222545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:54.222428    5718 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:04:56.341323 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:04:56.341296    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-66f6f89fb8-2wbfh"
Oct 31 01:05:00.051504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.051468    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b4bd988b-fsvnz"]
Oct 31 01:05:00.123946 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.123911    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.152712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.152675    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b4bd988b-fsvnz"]
Oct 31 01:05:00.286545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286504    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-config\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286555    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7babf137-b15f-4d1b-8963-dd5eed5250f0-tmp\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286572    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-trusted-ca-bundle\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286747    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-oauth-config\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286781    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-serving-cert\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286917 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286799    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-service-ca\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286915    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-oauth-serving-cert\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.286950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.286940    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdh2\" (UniqueName: \"kubernetes.io/projected/7babf137-b15f-4d1b-8963-dd5eed5250f0-kube-api-access-wzdh2\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.329821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.329741    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 01:05:00.388206 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388170    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-config\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388376 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388212    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7babf137-b15f-4d1b-8963-dd5eed5250f0-tmp\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388376 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388228    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-trusted-ca-bundle\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388376 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388263    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-oauth-config\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388376 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388291    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-serving-cert\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388376 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388317    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-service-ca\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388698 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388395    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-oauth-serving-cert\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388698 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388420    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdh2\" (UniqueName: \"kubernetes.io/projected/7babf137-b15f-4d1b-8963-dd5eed5250f0-kube-api-access-wzdh2\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388711    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7babf137-b15f-4d1b-8963-dd5eed5250f0-tmp\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.388934 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.388907    5718 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:05:00.389104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.389081    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-config\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.389230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.389206    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-oauth-serving-cert\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.389292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.389231    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-service-ca\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.389292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.389241    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7babf137-b15f-4d1b-8963-dd5eed5250f0-trusted-ca-bundle\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.391394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.391367    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-oauth-config\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.391394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.391366    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7babf137-b15f-4d1b-8963-dd5eed5250f0-console-serving-cert\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.447643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.447599    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdh2\" (UniqueName: \"kubernetes.io/projected/7babf137-b15f-4d1b-8963-dd5eed5250f0-kube-api-access-wzdh2\") pod \"console-9b4bd988b-fsvnz\" (UID: \"7babf137-b15f-4d1b-8963-dd5eed5250f0\") " pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:00.747146 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:00.747107    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:01.153092 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:01.152999    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b4bd988b-fsvnz"]
Oct 31 01:05:01.156007 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:05:01.155963    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7babf137_b15f_4d1b_8963_dd5eed5250f0.slice/crio-fa39064d106adc87bcb511e9b1c2188a9345841111d170efcb42d9023ed4be86 WatchSource:0}: Error finding container fa39064d106adc87bcb511e9b1c2188a9345841111d170efcb42d9023ed4be86: Status 404 returned error can't find the container with id fa39064d106adc87bcb511e9b1c2188a9345841111d170efcb42d9023ed4be86
Oct 31 01:05:02.115905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:02.115873    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b4bd988b-fsvnz" event={"ID":"7babf137-b15f-4d1b-8963-dd5eed5250f0","Type":"ContainerStarted","Data":"474d1d4d17c639a5af6257b3d7ed1e3b40d6a5e714b86982c390a553b7df988a"}
Oct 31 01:05:02.116081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:02.115922    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b4bd988b-fsvnz" event={"ID":"7babf137-b15f-4d1b-8963-dd5eed5250f0","Type":"ContainerStarted","Data":"fa39064d106adc87bcb511e9b1c2188a9345841111d170efcb42d9023ed4be86"}
Oct 31 01:05:02.316264 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:02.316186    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9b4bd988b-fsvnz" podStartSLOduration=2.3161703129999998 podStartE2EDuration="2.316170313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:02.267744413 +0000 UTC m=+18.430887827" watchObservedRunningTime="2026-10-31 01:05:02.316170313 +0000 UTC m=+18.479313725"
Oct 31 01:05:05.718329 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:05.718290    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6h6v"
Oct 31 01:05:07.923174 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:05:07.923138    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:05:09.164709 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:09.164677    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm9cp"
Oct 31 01:05:10.747942 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.747902    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:10.748328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.748050    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:10.752829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.752805    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:10.922754 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.922715    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:05:10.923170 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.923078    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="prometheus" containerID="cri-o://7fbe6480290e3dc5d85c9552c1c9f9c5620a3b482c0998dc1f78ddf87ab7b930" gracePeriod=600
Oct 31 01:05:10.923362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.923197    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://05d9c4bbca7496bc20f44a581d1afa8133d1113e2d8964905e993ce34d9feb3b" gracePeriod=600
Oct 31 01:05:10.923362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.923240    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-web" containerID="cri-o://e91dc4a49827f2eeb09a52cde1324c7d779092a8195c9e6f13a9ae99352a372b" gracePeriod=600
Oct 31 01:05:10.923362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.923107    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy" containerID="cri-o://43e35f7873b5472f643e0f99b9dc03181d4715eb079cbfc948db13c14057ef72" gracePeriod=600
Oct 31 01:05:10.923362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.923337    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="config-reloader" containerID="cri-o://72e0230c5bb4494cedbfc9fc957f9b4ff2c95c52336b4270e134e9ab8244a07d" gracePeriod=600
Oct 31 01:05:10.923714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:10.923331    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="thanos-sidecar" containerID="cri-o://f5fcf1217036a49bfc3bf12b3ec9e6f5e130c9250567c7045e1369bd4bf86486" gracePeriod=600
Oct 31 01:05:11.151142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151112    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="05d9c4bbca7496bc20f44a581d1afa8133d1113e2d8964905e993ce34d9feb3b" exitCode=0
Oct 31 01:05:11.151142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151137    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="43e35f7873b5472f643e0f99b9dc03181d4715eb079cbfc948db13c14057ef72" exitCode=0
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151151    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="e91dc4a49827f2eeb09a52cde1324c7d779092a8195c9e6f13a9ae99352a372b" exitCode=0
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151158    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="f5fcf1217036a49bfc3bf12b3ec9e6f5e130c9250567c7045e1369bd4bf86486" exitCode=0
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151164    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="72e0230c5bb4494cedbfc9fc957f9b4ff2c95c52336b4270e134e9ab8244a07d" exitCode=0
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151173    5718 generic.go:356] "Generic (PLEG): container finished" podID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerID="7fbe6480290e3dc5d85c9552c1c9f9c5620a3b482c0998dc1f78ddf87ab7b930" exitCode=0
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151182    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"05d9c4bbca7496bc20f44a581d1afa8133d1113e2d8964905e993ce34d9feb3b"}
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151253    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"43e35f7873b5472f643e0f99b9dc03181d4715eb079cbfc948db13c14057ef72"}
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151284    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"e91dc4a49827f2eeb09a52cde1324c7d779092a8195c9e6f13a9ae99352a372b"}
Oct 31 01:05:11.151337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151315    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"f5fcf1217036a49bfc3bf12b3ec9e6f5e130c9250567c7045e1369bd4bf86486"}
Oct 31 01:05:11.151596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151354    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"72e0230c5bb4494cedbfc9fc957f9b4ff2c95c52336b4270e134e9ab8244a07d"}
Oct 31 01:05:11.151596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.151387    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"7fbe6480290e3dc5d85c9552c1c9f9c5620a3b482c0998dc1f78ddf87ab7b930"}
Oct 31 01:05:11.155219 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.155204    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b4bd988b-fsvnz"
Oct 31 01:05:11.180006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.179986    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:11.188251 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.188226    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.188386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.188279    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.188386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.188338    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.188509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.188426    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.188569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.188509    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.188623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.188588    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.189281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.189254    5718 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.189496 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.189459    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:11.189924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.189895    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.190889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.190864    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:11.190964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.190926    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.290339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290261    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290302    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290329    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290350    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290373    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290415    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290434    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290458    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290490    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290511    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290531    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290563    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290589    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290606    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.290616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290621    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") pod \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\" (UID: \"dbfe986f-2a26-4c74-9564-dbb4585d53cd\") "
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290776    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290893    5718 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290914    5718 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290930    5718 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-tls-assets\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290945    5718 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-db\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290960    5718 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.291386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.290994    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.291781 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.291408    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.292055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.291871    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.292055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.292010    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:11.296068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.296042    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.296261 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.296235    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "kube-api-access-gw6tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:11.302092 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.302062    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.302356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.302103    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:11.302356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.302174    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.302356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.302245    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.302356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.302329    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.303102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.303071    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.303195 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.303128    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.311865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.311821    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config" pod "dbfe986f-2a26-4c74-9564-dbb4585d53cd" (UID: "dbfe986f-2a26-4c74-9564-dbb4585d53cd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:11.392161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392124    5718 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392154    5718 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392165    5718 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-metrics-client-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392176    5718 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-grpc-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392187    5718 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dbfe986f-2a26-4c74-9564-dbb4585d53cd-config-out\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392195    5718 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392205    5718 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-kube-rbac-proxy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392213    5718 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392223    5718 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392232    5718 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-web-config\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392242    5718 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392252    5718 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbfe986f-2a26-4c74-9564-dbb4585d53cd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392261    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw6tt\" (UniqueName: \"kubernetes.io/projected/dbfe986f-2a26-4c74-9564-dbb4585d53cd-kube-api-access-gw6tt\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:11.392408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:11.392269    5718 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dbfe986f-2a26-4c74-9564-dbb4585d53cd-secret-metrics-client-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:12.157835 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.157802    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.157835 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.157802    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dbfe986f-2a26-4c74-9564-dbb4585d53cd","Type":"ContainerDied","Data":"8a701991fc0e1dc2e8567866b492c5e26b3b7b063b59b8f2d9a4df4d696fab49"}
Oct 31 01:05:12.158362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.157907    5718 scope.go:122] "RemoveContainer" containerID="05d9c4bbca7496bc20f44a581d1afa8133d1113e2d8964905e993ce34d9feb3b"
Oct 31 01:05:12.165975 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.165951    5718 scope.go:122] "RemoveContainer" containerID="43e35f7873b5472f643e0f99b9dc03181d4715eb079cbfc948db13c14057ef72"
Oct 31 01:05:12.173193 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.173172    5718 scope.go:122] "RemoveContainer" containerID="e91dc4a49827f2eeb09a52cde1324c7d779092a8195c9e6f13a9ae99352a372b"
Oct 31 01:05:12.180074 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.180057    5718 scope.go:122] "RemoveContainer" containerID="f5fcf1217036a49bfc3bf12b3ec9e6f5e130c9250567c7045e1369bd4bf86486"
Oct 31 01:05:12.186722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.186701    5718 scope.go:122] "RemoveContainer" containerID="72e0230c5bb4494cedbfc9fc957f9b4ff2c95c52336b4270e134e9ab8244a07d"
Oct 31 01:05:12.193365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.193346    5718 scope.go:122] "RemoveContainer" containerID="7fbe6480290e3dc5d85c9552c1c9f9c5620a3b482c0998dc1f78ddf87ab7b930"
Oct 31 01:05:12.200746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.200722    5718 scope.go:122] "RemoveContainer" containerID="59b46adfead674c24e08869acb4255dd10bb6fdf2c26c6a0462c018cc49d1f80"
Oct 31 01:05:12.379573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.379533    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:05:12.428998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.428967    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:05:12.429317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429296    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:05:12.429592 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429581    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="thanos-sidecar"
Oct 31 01:05:12.429647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429595    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="thanos-sidecar"
Oct 31 01:05:12.429647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429611    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="config-reloader"
Oct 31 01:05:12.429647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429616    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="config-reloader"
Oct 31 01:05:12.429647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429635    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-thanos"
Oct 31 01:05:12.429647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429641    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-thanos"
Oct 31 01:05:12.429647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429663    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="prometheus"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429669    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="prometheus"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429679    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-web"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429684    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-web"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429692    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="init-config-reloader"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429696    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="init-config-reloader"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429702    5718 cpu_manager.go:414] "containerMap: removing container" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429706    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429749    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="thanos-sidecar"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429756    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="config-reloader"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429764    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-thanos"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429769    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="prometheus"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429776    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy-web"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429782    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="init-config-reloader"
Oct 31 01:05:12.429844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.429787    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" containerName="kube-rbac-proxy"
Oct 31 01:05:12.433471 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.433450    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.487712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.487681    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 01:05:12.487977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.487952    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 01:05:12.487977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.487952    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 01:05:12.487977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.487956    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488032    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488048    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488071    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488087    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488086    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488105    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 01:05:12.488266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488181    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zmtbq\""
Oct 31 01:05:12.488757 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488743    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 01:05:12.488826 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.488815    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsmab2l1uuhg0\""
Oct 31 01:05:12.500261 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500235    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500381 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500268    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500381 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500290    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500381 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500356    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500475 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500386    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-config-out\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500475 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500410    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500536 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500491    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500536 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500511    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-config\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500536 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500528    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500627 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500542    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-web-config\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500625    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500647    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500755 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500692    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500755 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500733    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500755 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500751    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500770    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500788    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500809    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500837    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.500923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.500862    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgnl\" (UniqueName: \"kubernetes.io/projected/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-kube-api-access-rlgnl\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.538902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.538872    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 01:05:12.539252 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.539236    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 01:05:12.539359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.539342    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 01:05:12.601733 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601687    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.601733 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601736    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.601974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601832    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.601974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601870    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.601974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601889    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.601974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601908    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.601974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601937    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.601975    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602005    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602027    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgnl\" (UniqueName: \"kubernetes.io/projected/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-kube-api-access-rlgnl\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602072    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602109    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602133    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602159    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602189    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-config-out\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602221    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602282    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602304    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-config\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602334    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602359    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-web-config\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.602790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.602786    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.604459 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.603573    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.604459 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.604122    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.605160 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.605056    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.605706 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.605683    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.605849 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.605820    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.606042 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.606019    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.606705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.606194    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.606705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.606521    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.606705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.606593    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-web-config\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.606705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.606629    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.606959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.606740    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.607366 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.607275    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.607366 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.607295    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.607574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.607555    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.607955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.607934    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.608596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.608578    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.608810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.608795    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-config-out\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.608929 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.608913    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-config\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.650922 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.650886    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:05:12.661986 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.661958    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgnl\" (UniqueName: \"kubernetes.io/projected/d90e5bc8-ff48-4edc-9a2e-16496cc7c497-kube-api-access-rlgnl\") pod \"prometheus-k8s-0\" (UID: \"d90e5bc8-ff48-4edc-9a2e-16496cc7c497\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:12.750666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:12.750617    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:13.177909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:13.177832    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 01:05:13.180627 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:05:13.180597    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90e5bc8_ff48_4edc_9a2e_16496cc7c497.slice/crio-285e6543e897acc2cde25e3daaba73d26012e2d0ec4a678266ad9c1381c00e59 WatchSource:0}: Error finding container 285e6543e897acc2cde25e3daaba73d26012e2d0ec4a678266ad9c1381c00e59: Status 404 returned error can't find the container with id 285e6543e897acc2cde25e3daaba73d26012e2d0ec4a678266ad9c1381c00e59
Oct 31 01:05:13.964909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:13.964878    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfe986f-2a26-4c74-9564-dbb4585d53cd" path="/var/lib/kubelet/pods/dbfe986f-2a26-4c74-9564-dbb4585d53cd/volumes"
Oct 31 01:05:14.167462 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:14.167419    5718 generic.go:356] "Generic (PLEG): container finished" podID="d90e5bc8-ff48-4edc-9a2e-16496cc7c497" containerID="cc5423c5e1db30db1ed8c225a8db9d5579f2ee68539f692b45c10cbd92101f7b" exitCode=0
Oct 31 01:05:14.167640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:14.167489    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerDied","Data":"cc5423c5e1db30db1ed8c225a8db9d5579f2ee68539f692b45c10cbd92101f7b"}
Oct 31 01:05:14.167640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:14.167580    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"285e6543e897acc2cde25e3daaba73d26012e2d0ec4a678266ad9c1381c00e59"}
Oct 31 01:05:15.176066 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.176016    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"4fd9fcdb2db45c2a8c0d8539acdefb537b4d7d6c20677232a661f5a5eabb236f"}
Oct 31 01:05:15.176603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.176092    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"2b9ae52b3135d37b458f0a2941c58b0ca1d6198f189ee721d33db09781526471"}
Oct 31 01:05:15.176603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.176133    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"794ba2751fd29b6e03226b023606babd90f10b9fa5c442c46b1e0d743902eb43"}
Oct 31 01:05:15.176603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.176164    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"813e7aa9e5702e501ef497270c0c8062c5b0b1082ae3fc1077a6566ad77cc136"}
Oct 31 01:05:15.176603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.176189    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"d6f42bf1ca9eb7ef6a2195ee35344a491c24ee0f87cf35e1015a2b438386a59b"}
Oct 31 01:05:15.176603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.176211    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d90e5bc8-ff48-4edc-9a2e-16496cc7c497","Type":"ContainerStarted","Data":"798db11ceefdc6b6e0103b33461f8bcbca30d239fc35b562723190356cee8280"}
Oct 31 01:05:15.309893 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:15.309834    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.288010917 podStartE2EDuration="3.309820286s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.809369ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:15.308987689 +0000 UTC m=+31.472131128" watchObservedRunningTime="2026-10-31 01:05:15.309820286 +0000 UTC m=+31.472963700"
Oct 31 01:05:17.750801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:17.750754    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:05:18.160219 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.160127    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b5d58f5bd-vchll" podUID="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" containerName="console" containerID="cri-o://135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630" gracePeriod=15
Oct 31 01:05:18.428979 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.428951    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:05:18.456355 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456328    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456362    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456384    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456408    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456592 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456467    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456592 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456488    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456592 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456518    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") pod \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\" (UID: \"7bb40088-25e1-47b8-823a-f6ba2ef3e71d\") "
Oct 31 01:05:18.456878 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456850    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:18.457002 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456964    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:18.457002 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456980    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:18.457134 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.456844    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:18.459016 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.458994    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "kube-api-access-rctgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:18.459200 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.459177    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:18.459246 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.459222    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert" pod "7bb40088-25e1-47b8-823a-f6ba2ef3e71d" (UID: "7bb40088-25e1-47b8-823a-f6ba2ef3e71d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:18.557886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557851    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rctgq\" (UniqueName: \"kubernetes.io/projected/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-kube-api-access-rctgq\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:18.557886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557882    5718 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-oauth-serving-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:18.557886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557894    5718 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-tmp\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:18.558116 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557903    5718 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-serving-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:18.558116 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557917    5718 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-config\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:18.558116 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557931    5718 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:18.558116 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:18.557939    5718 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7bb40088-25e1-47b8-823a-f6ba2ef3e71d-console-oauth-config\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:19.190410 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.190377    5718 generic.go:356] "Generic (PLEG): container finished" podID="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" containerID="135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630" exitCode=0
Oct 31 01:05:19.190843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.190426    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d58f5bd-vchll"
Oct 31 01:05:19.190843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.190442    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d58f5bd-vchll" event={"ID":"7bb40088-25e1-47b8-823a-f6ba2ef3e71d","Type":"ContainerDied","Data":"135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630"}
Oct 31 01:05:19.190843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.190483    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d58f5bd-vchll" event={"ID":"7bb40088-25e1-47b8-823a-f6ba2ef3e71d","Type":"ContainerDied","Data":"aa942a4855493ac5522e5ebcb192906917084578234c0e7827ce7e56b95da544"}
Oct 31 01:05:19.190843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.190502    5718 scope.go:122] "RemoveContainer" containerID="135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630"
Oct 31 01:05:19.202285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.202258    5718 scope.go:122] "RemoveContainer" containerID="135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630"
Oct 31 01:05:19.202731 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:05:19.202706    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630\": container with ID starting with 135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630 not found: ID does not exist" containerID="135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630"
Oct 31 01:05:19.202826 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.202741    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630"} err="failed to get container status \"135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630\": rpc error: code = NotFound desc = could not find container \"135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630\": container with ID starting with 135f480809d2b67c7b5dd13956c84f43faf0a8b7bc8d43635ee8898b93f46630 not found: ID does not exist"
Oct 31 01:05:19.358148 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.358110    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5d58f5bd-vchll"]
Oct 31 01:05:19.408281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.408252    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b5d58f5bd-vchll"]
Oct 31 01:05:19.964596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:19.964557    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" path="/var/lib/kubelet/pods/7bb40088-25e1-47b8-823a-f6ba2ef3e71d/volumes"
Oct 31 01:05:21.530443 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:05:21.530411    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fczdh"
Oct 31 01:05:37.923404 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:05:37.923362    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:06:02.410465 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:06:02.410421    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:06:02.429118 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:06:02.429087    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:06:02.767585 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:06:02.767558    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 01:06:07.922973 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:06:07.922924    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:06:37.923473 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:06:37.923429    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:07:07.923418 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:07:07.923377    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:07:20.259345 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.259309    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"]
Oct 31 01:07:20.259828 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.259610    5718 cpu_manager.go:414] "containerMap: removing container" podUID="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" containerName="console"
Oct 31 01:07:20.259828 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.259619    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" containerName="console"
Oct 31 01:07:20.259828 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.259685    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb40088-25e1-47b8-823a-f6ba2ef3e71d" containerName="console"
Oct 31 01:07:20.262694 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.262674    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.366495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.366464    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:07:20.366712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.366636    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:07:20.366712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.366690    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-4wkzc\""
Oct 31 01:07:20.367391 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.367371    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"]
Oct 31 01:07:20.383924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.383896    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.384081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.383936    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vts4f\" (UniqueName: \"kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.384121 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.384079    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.484622 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.484586    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.484622 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.484623    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.484890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.484672    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts4f\" (UniqueName: \"kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.485025 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.485001    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.485099 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.485053    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.542699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.542596    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vts4f\" (UniqueName: \"kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.578521 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.578484    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:20.958637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.958613    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"]
Oct 31 01:07:20.960759 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:07:20.960734    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5fa42d_7167_452c_ae33_3680958b8456.slice/crio-a8784a3b7315671e9c6040880061e44667caea2e7d94c1867600efdc17da98c7 WatchSource:0}: Error finding container a8784a3b7315671e9c6040880061e44667caea2e7d94c1867600efdc17da98c7: Status 404 returned error can't find the container with id a8784a3b7315671e9c6040880061e44667caea2e7d94c1867600efdc17da98c7
Oct 31 01:07:20.962830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:20.962799    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:07:21.633271 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.633240    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" event={"ID":"6f5fa42d-7167-452c-ae33-3680958b8456","Type":"ContainerStarted","Data":"a8784a3b7315671e9c6040880061e44667caea2e7d94c1867600efdc17da98c7"}
Oct 31 01:07:21.879496 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.879456    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"]
Oct 31 01:07:21.883510 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.883455    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:21.987364 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.987329    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-k56tr\""
Oct 31 01:07:21.988192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.988168    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"]
Oct 31 01:07:21.988739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.988719    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"]
Oct 31 01:07:21.992279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.992258    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:21.999238 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.999214    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhk6\" (UniqueName: \"kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:21.999496 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.999275    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:21.999496 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:21.999398    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.046210 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.046178    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"]
Oct 31 01:07:22.051721 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.051694    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-tlg2v\""
Oct 31 01:07:22.101104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101067    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.101295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101143    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.101295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101174    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhk6\" (UniqueName: \"kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.101295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101225    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.101460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101316    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.101460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101370    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5srz\" (UniqueName: \"kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.101580 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101490    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.101639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.101613    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.160929 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.160851    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhk6\" (UniqueName: \"kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.202102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.201796    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:22.202102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.201981    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.202102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.202026    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.202102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.202071    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5srz\" (UniqueName: \"kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.202830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.202791    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.203136 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.203117    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.266202 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.266175    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5srz\" (UniqueName: \"kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.311206 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.311163    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:22.583466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.583437    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"]
Oct 31 01:07:22.586169 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:07:22.586138    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f28295_f390_4b2d_b5b3_f5257a8a6a65.slice/crio-20044ede78d8f43988ff9233a4bf93b9fcbe9d58531c9e32edd0339ea01de60f WatchSource:0}: Error finding container 20044ede78d8f43988ff9233a4bf93b9fcbe9d58531c9e32edd0339ea01de60f: Status 404 returned error can't find the container with id 20044ede78d8f43988ff9233a4bf93b9fcbe9d58531c9e32edd0339ea01de60f
Oct 31 01:07:22.639680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.639624    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2" event={"ID":"b7f28295-f390-4b2d-b5b3-f5257a8a6a65","Type":"ContainerStarted","Data":"20044ede78d8f43988ff9233a4bf93b9fcbe9d58531c9e32edd0339ea01de60f"}
Oct 31 01:07:22.691885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:22.691848    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"]
Oct 31 01:07:22.696089 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:07:22.696057    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84c24af_6b5f_481c_b35e_0217678dc610.slice/crio-37bd74aabd80abf7c35e4958d64da23fe4b1187be7a7d51c66de98f9c4cfa8f0 WatchSource:0}: Error finding container 37bd74aabd80abf7c35e4958d64da23fe4b1187be7a7d51c66de98f9c4cfa8f0: Status 404 returned error can't find the container with id 37bd74aabd80abf7c35e4958d64da23fe4b1187be7a7d51c66de98f9c4cfa8f0
Oct 31 01:07:23.645889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:23.645849    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk" event={"ID":"a84c24af-6b5f-481c-b35e-0217678dc610","Type":"ContainerStarted","Data":"37bd74aabd80abf7c35e4958d64da23fe4b1187be7a7d51c66de98f9c4cfa8f0"}
Oct 31 01:07:26.287864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.287820    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"]
Oct 31 01:07:26.293539 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.293512    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.390826 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.390785    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:07:26.390826 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.390805    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-2bhrz\""
Oct 31 01:07:26.390826 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.390807    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:07:26.399931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.399906    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:07:26.400895 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.400869    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"]
Oct 31 01:07:26.444233 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444191    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t8g5\" (UniqueName: \"kubernetes.io/projected/96a77a21-5cc4-4025-bd5c-4c2492aac124-kube-api-access-7t8g5\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444277    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444343    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444498 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444434    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444498 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444466    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a77a21-5cc4-4025-bd5c-4c2492aac124-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444498 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444491    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444577    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444608    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.444737 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.444634    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545469    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545526    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545563    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545592    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a77a21-5cc4-4025-bd5c-4c2492aac124-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545621    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545683    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545730    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.545808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545769    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.546023 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.545827    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t8g5\" (UniqueName: \"kubernetes.io/projected/96a77a21-5cc4-4025-bd5c-4c2492aac124-kube-api-access-7t8g5\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.546713 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.546642    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.546713 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.546702    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.547142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.547102    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.547305 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.547111    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.547509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.547490    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a77a21-5cc4-4025-bd5c-4c2492aac124-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.550088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.550055    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.550323 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.550294    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.609113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.608609    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a77a21-5cc4-4025-bd5c-4c2492aac124-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.609113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.609051    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t8g5\" (UniqueName: \"kubernetes.io/projected/96a77a21-5cc4-4025-bd5c-4c2492aac124-kube-api-access-7t8g5\") pod \"openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks\" (UID: \"96a77a21-5cc4-4025-bd5c-4c2492aac124\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:26.612780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:26.612749    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:27.061690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:27.061560    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"]
Oct 31 01:07:27.067293 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:07:27.067259    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a77a21_5cc4_4025_bd5c_4c2492aac124.slice/crio-e53e42a6daa371af04473846407538224a0508b50f533f8b6a46f23b7d6ccea4 WatchSource:0}: Error finding container e53e42a6daa371af04473846407538224a0508b50f533f8b6a46f23b7d6ccea4: Status 404 returned error can't find the container with id e53e42a6daa371af04473846407538224a0508b50f533f8b6a46f23b7d6ccea4
Oct 31 01:07:27.671455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:27.671420    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks" event={"ID":"96a77a21-5cc4-4025-bd5c-4c2492aac124","Type":"ContainerStarted","Data":"e53e42a6daa371af04473846407538224a0508b50f533f8b6a46f23b7d6ccea4"}
Oct 31 01:07:28.677645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:28.677559    5718 generic.go:356] "Generic (PLEG): container finished" podID="6f5fa42d-7167-452c-ae33-3680958b8456" containerID="402edfe411aec3cdb82dabd662eacdc66bb26424bf37fda7568d468d58ddbd5f" exitCode=0
Oct 31 01:07:28.677645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:28.677637    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" event={"ID":"6f5fa42d-7167-452c-ae33-3680958b8456","Type":"ContainerDied","Data":"402edfe411aec3cdb82dabd662eacdc66bb26424bf37fda7568d468d58ddbd5f"}
Oct 31 01:07:28.680155 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:28.680117    5718 generic.go:356] "Generic (PLEG): container finished" podID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerID="ca9fb9701989f8499fe1d8b38b3b5dcd788a513ffc05610b4421d20e4ae12a08" exitCode=0
Oct 31 01:07:28.680253 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:28.680212    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2" event={"ID":"b7f28295-f390-4b2d-b5b3-f5257a8a6a65","Type":"ContainerDied","Data":"ca9fb9701989f8499fe1d8b38b3b5dcd788a513ffc05610b4421d20e4ae12a08"}
Oct 31 01:07:30.692159 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:30.692057    5718 generic.go:356] "Generic (PLEG): container finished" podID="a84c24af-6b5f-481c-b35e-0217678dc610" containerID="435f03435ed20e35f21f4d2416c18a8d5d706f8e732996e21821df082e0d6ced" exitCode=0
Oct 31 01:07:30.692645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:30.692215    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk" event={"ID":"a84c24af-6b5f-481c-b35e-0217678dc610","Type":"ContainerDied","Data":"435f03435ed20e35f21f4d2416c18a8d5d706f8e732996e21821df082e0d6ced"}
Oct 31 01:07:30.694412 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:30.694372    5718 generic.go:356] "Generic (PLEG): container finished" podID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerID="f54b78adc47b299ad47f15aac57f4b3a1308f0fabd01361fe91c489975434e95" exitCode=0
Oct 31 01:07:30.694528 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:30.694413    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2" event={"ID":"b7f28295-f390-4b2d-b5b3-f5257a8a6a65","Type":"ContainerDied","Data":"f54b78adc47b299ad47f15aac57f4b3a1308f0fabd01361fe91c489975434e95"}
Oct 31 01:07:31.278692 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:31.278633    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:07:31.278824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:31.278748    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:07:31.278824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:31.278784    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:07:31.701819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:31.701712    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks" event={"ID":"96a77a21-5cc4-4025-bd5c-4c2492aac124","Type":"ContainerStarted","Data":"b2ae04e9c9f4f9ca041f7ce340515b7a354c07d8be00caef683d9d27e99ad988"}
Oct 31 01:07:31.873974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:31.873907    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks" podStartSLOduration=1.667790644 podStartE2EDuration="5.873887419s" totalImagesPullingTime="4.206096775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:31.872913475 +0000 UTC m=+168.036056893" watchObservedRunningTime="2026-10-31 01:07:31.873887419 +0000 UTC m=+168.037030836"
Oct 31 01:07:32.613982 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:32.613945    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:32.620086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:32.619849    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:32.709211 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:32.709180    5718 generic.go:356] "Generic (PLEG): container finished" podID="6f5fa42d-7167-452c-ae33-3680958b8456" containerID="28628b0f44feb9c802d273faf37514cc73ce1a86653d478817cce34e508efbca" exitCode=0
Oct 31 01:07:32.709641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:32.709244    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" event={"ID":"6f5fa42d-7167-452c-ae33-3680958b8456","Type":"ContainerDied","Data":"28628b0f44feb9c802d273faf37514cc73ce1a86653d478817cce34e508efbca"}
Oct 31 01:07:32.709734 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:32.709703    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:32.710888 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:32.710870    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks"
Oct 31 01:07:33.717647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:33.717259    5718 generic.go:356] "Generic (PLEG): container finished" podID="a84c24af-6b5f-481c-b35e-0217678dc610" containerID="682b95c6330bd1677a3f5340f14dc4564a473ae7d7e8b0acce06c0c93f239e7f" exitCode=0
Oct 31 01:07:33.718364 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:33.718328    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk" event={"ID":"a84c24af-6b5f-481c-b35e-0217678dc610","Type":"ContainerDied","Data":"682b95c6330bd1677a3f5340f14dc4564a473ae7d7e8b0acce06c0c93f239e7f"}
Oct 31 01:07:37.935814 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:07:37.935743    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:07:39.751511 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:39.751465    5718 generic.go:356] "Generic (PLEG): container finished" podID="a84c24af-6b5f-481c-b35e-0217678dc610" containerID="af0c63456021e0d09927157a64a142521be7eb0f337e173b980d0c1a67267511" exitCode=0
Oct 31 01:07:39.751955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:39.751549    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk" event={"ID":"a84c24af-6b5f-481c-b35e-0217678dc610","Type":"ContainerDied","Data":"af0c63456021e0d09927157a64a142521be7eb0f337e173b980d0c1a67267511"}
Oct 31 01:07:39.754701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:39.754546    5718 generic.go:356] "Generic (PLEG): container finished" podID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerID="5ee2419fa2bd208618d20e0c4d79e665082070179c521f4bcdbcf9b899df569e" exitCode=0
Oct 31 01:07:39.754701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:39.754624    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2" event={"ID":"b7f28295-f390-4b2d-b5b3-f5257a8a6a65","Type":"ContainerDied","Data":"5ee2419fa2bd208618d20e0c4d79e665082070179c521f4bcdbcf9b899df569e"}
Oct 31 01:07:40.760278 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.760237    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" event={"ID":"6f5fa42d-7167-452c-ae33-3680958b8456","Type":"ContainerStarted","Data":"e96e0e40f7135e3d158f16b1cc90ecc2cb9a2f7a6c5f1d3e3d0680257b354498"}
Oct 31 01:07:40.913538 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.913511    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:40.919457 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.919435    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:40.928943 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.928897    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" podStartSLOduration=2.369715419 podStartE2EDuration="20.928883507s" totalImagesPullingTime="18.559168088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:40.928073777 +0000 UTC m=+177.091217191" watchObservedRunningTime="2026-10-31 01:07:40.928883507 +0000 UTC m=+177.092026921"
Oct 31 01:07:40.987155 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987121    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util\") pod \"a84c24af-6b5f-481c-b35e-0217678dc610\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") "
Oct 31 01:07:40.987341 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987180    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle\") pod \"a84c24af-6b5f-481c-b35e-0217678dc610\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") "
Oct 31 01:07:40.987341 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987206    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle\") pod \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") "
Oct 31 01:07:40.987341 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987239    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util\") pod \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") "
Oct 31 01:07:40.987341 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987285    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz\" (UniqueName: \"kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz\") pod \"a84c24af-6b5f-481c-b35e-0217678dc610\" (UID: \"a84c24af-6b5f-481c-b35e-0217678dc610\") "
Oct 31 01:07:40.987341 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987318    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6\" (UniqueName: \"kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6\") pod \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\" (UID: \"b7f28295-f390-4b2d-b5b3-f5257a8a6a65\") "
Oct 31 01:07:40.987839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987805    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle" pod "a84c24af-6b5f-481c-b35e-0217678dc610" (UID: "a84c24af-6b5f-481c-b35e-0217678dc610"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:40.987839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.987823    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle" pod "b7f28295-f390-4b2d-b5b3-f5257a8a6a65" (UID: "b7f28295-f390-4b2d-b5b3-f5257a8a6a65"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:40.990387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.990361    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz" pod "a84c24af-6b5f-481c-b35e-0217678dc610" (UID: "a84c24af-6b5f-481c-b35e-0217678dc610"). InnerVolumeSpecName "kube-api-access-t5srz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:40.990387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.990370    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6" pod "b7f28295-f390-4b2d-b5b3-f5257a8a6a65" (UID: "b7f28295-f390-4b2d-b5b3-f5257a8a6a65"). InnerVolumeSpecName "kube-api-access-wzhk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:40.991692 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.991671    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util" pod "b7f28295-f390-4b2d-b5b3-f5257a8a6a65" (UID: "b7f28295-f390-4b2d-b5b3-f5257a8a6a65"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:40.992190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:40.992165    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util" pod "a84c24af-6b5f-481c-b35e-0217678dc610" (UID: "a84c24af-6b5f-481c-b35e-0217678dc610"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:41.088491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.088455    5718 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-util\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.088491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.088486    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5srz\" (UniqueName: \"kubernetes.io/projected/a84c24af-6b5f-481c-b35e-0217678dc610-kube-api-access-t5srz\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.088491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.088495    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzhk6\" (UniqueName: \"kubernetes.io/projected/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-kube-api-access-wzhk6\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.088491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.088504    5718 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-util\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.088807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.088513    5718 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a84c24af-6b5f-481c-b35e-0217678dc610-bundle\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.088807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.088523    5718 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7f28295-f390-4b2d-b5b3-f5257a8a6a65-bundle\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.765701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.765671    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk"
Oct 31 01:07:41.766129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.765695    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hz4pk" event={"ID":"a84c24af-6b5f-481c-b35e-0217678dc610","Type":"ContainerDied","Data":"37bd74aabd80abf7c35e4958d64da23fe4b1187be7a7d51c66de98f9c4cfa8f0"}
Oct 31 01:07:41.766129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.765747    5718 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bd74aabd80abf7c35e4958d64da23fe4b1187be7a7d51c66de98f9c4cfa8f0"
Oct 31 01:07:41.767843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.767818    5718 generic.go:356] "Generic (PLEG): container finished" podID="6f5fa42d-7167-452c-ae33-3680958b8456" containerID="e96e0e40f7135e3d158f16b1cc90ecc2cb9a2f7a6c5f1d3e3d0680257b354498" exitCode=0
Oct 31 01:07:41.767971 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.767888    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" event={"ID":"6f5fa42d-7167-452c-ae33-3680958b8456","Type":"ContainerDied","Data":"e96e0e40f7135e3d158f16b1cc90ecc2cb9a2f7a6c5f1d3e3d0680257b354498"}
Oct 31 01:07:41.769606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.769587    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2" event={"ID":"b7f28295-f390-4b2d-b5b3-f5257a8a6a65","Type":"ContainerDied","Data":"20044ede78d8f43988ff9233a4bf93b9fcbe9d58531c9e32edd0339ea01de60f"}
Oct 31 01:07:41.769719 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.769617    5718 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20044ede78d8f43988ff9233a4bf93b9fcbe9d58531c9e32edd0339ea01de60f"
Oct 31 01:07:41.769719 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:41.769627    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crldn2"
Oct 31 01:07:42.977864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:42.977829    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:43.003889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.003858    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util\") pod \"6f5fa42d-7167-452c-ae33-3680958b8456\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") "
Oct 31 01:07:43.004010 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.003949    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f\" (UniqueName: \"kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f\") pod \"6f5fa42d-7167-452c-ae33-3680958b8456\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") "
Oct 31 01:07:43.004374 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.003993    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle\") pod \"6f5fa42d-7167-452c-ae33-3680958b8456\" (UID: \"6f5fa42d-7167-452c-ae33-3680958b8456\") "
Oct 31 01:07:43.010957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.004989    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle" pod "6f5fa42d-7167-452c-ae33-3680958b8456" (UID: "6f5fa42d-7167-452c-ae33-3680958b8456"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:43.010957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.010074    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f" pod "6f5fa42d-7167-452c-ae33-3680958b8456" (UID: "6f5fa42d-7167-452c-ae33-3680958b8456"). InnerVolumeSpecName "kube-api-access-vts4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:43.010957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.010272    5718 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-bundle\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:43.010957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.010304    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vts4f\" (UniqueName: \"kubernetes.io/projected/6f5fa42d-7167-452c-ae33-3680958b8456-kube-api-access-vts4f\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:43.011913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.011885    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util" pod "6f5fa42d-7167-452c-ae33-3680958b8456" (UID: "6f5fa42d-7167-452c-ae33-3680958b8456"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:43.110882 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.110795    5718 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f5fa42d-7167-452c-ae33-3680958b8456-util\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:43.779810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.779784    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm"
Oct 31 01:07:43.779977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.779780    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fv2pm" event={"ID":"6f5fa42d-7167-452c-ae33-3680958b8456","Type":"ContainerDied","Data":"a8784a3b7315671e9c6040880061e44667caea2e7d94c1867600efdc17da98c7"}
Oct 31 01:07:43.779977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:43.779924    5718 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8784a3b7315671e9c6040880061e44667caea2e7d94c1867600efdc17da98c7"
Oct 31 01:07:48.707488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707452    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"]
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707766    5718 cpu_manager.go:414] "containerMap: removing container" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="pull"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707777    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="pull"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707794    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="util"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707799    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="util"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707806    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="pull"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707811    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="pull"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707818    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707822    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707832    5718 cpu_manager.go:414] "containerMap: removing container" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="util"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707837    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="util"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707848    5718 cpu_manager.go:414] "containerMap: removing container" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707854    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707863    5718 cpu_manager.go:414] "containerMap: removing container" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="util"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707868    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="util"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707878    5718 cpu_manager.go:414] "containerMap: removing container" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707883    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707893    5718 cpu_manager.go:414] "containerMap: removing container" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="pull"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707897    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="pull"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707939    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84c24af-6b5f-481c-b35e-0217678dc610" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707947    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f28295-f390-4b2d-b5b3-f5257a8a6a65" containerName="extract"
Oct 31 01:07:48.707998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.707955    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5fa42d-7167-452c-ae33-3680958b8456" containerName="extract"
Oct 31 01:07:48.710061 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.710040    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:48.766082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.766045    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5hg\" (UniqueName: \"kubernetes.io/projected/46801ca3-13d4-4f46-8d06-662fa3633170-kube-api-access-nt5hg\") pod \"jobset-operator-674cd59dc5-ht5d9\" (UID: \"46801ca3-13d4-4f46-8d06-662fa3633170\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:48.766290 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.766177    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46801ca3-13d4-4f46-8d06-662fa3633170-tmp\") pod \"jobset-operator-674cd59dc5-ht5d9\" (UID: \"46801ca3-13d4-4f46-8d06-662fa3633170\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:48.815832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.815802    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-qdmnt\""
Oct 31 01:07:48.816024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.815845    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 01:07:48.816024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.815852    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:07:48.816881 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.816862    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"]
Oct 31 01:07:48.866738 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.866710    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5hg\" (UniqueName: \"kubernetes.io/projected/46801ca3-13d4-4f46-8d06-662fa3633170-kube-api-access-nt5hg\") pod \"jobset-operator-674cd59dc5-ht5d9\" (UID: \"46801ca3-13d4-4f46-8d06-662fa3633170\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:48.866953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.866794    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46801ca3-13d4-4f46-8d06-662fa3633170-tmp\") pod \"jobset-operator-674cd59dc5-ht5d9\" (UID: \"46801ca3-13d4-4f46-8d06-662fa3633170\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:48.867133 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.867115    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46801ca3-13d4-4f46-8d06-662fa3633170-tmp\") pod \"jobset-operator-674cd59dc5-ht5d9\" (UID: \"46801ca3-13d4-4f46-8d06-662fa3633170\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:48.925223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:48.925185    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5hg\" (UniqueName: \"kubernetes.io/projected/46801ca3-13d4-4f46-8d06-662fa3633170-kube-api-access-nt5hg\") pod \"jobset-operator-674cd59dc5-ht5d9\" (UID: \"46801ca3-13d4-4f46-8d06-662fa3633170\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:49.029770 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:49.029740    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"
Oct 31 01:07:49.398615 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:49.398535    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9"]
Oct 31 01:07:49.401777 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:07:49.401742    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46801ca3_13d4_4f46_8d06_662fa3633170.slice/crio-156eae2bb59b9553399c0c9d796c5fade87b2e8cea11a1f6ca8521367f29f051 WatchSource:0}: Error finding container 156eae2bb59b9553399c0c9d796c5fade87b2e8cea11a1f6ca8521367f29f051: Status 404 returned error can't find the container with id 156eae2bb59b9553399c0c9d796c5fade87b2e8cea11a1f6ca8521367f29f051
Oct 31 01:07:49.803385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:49.803340    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9" event={"ID":"46801ca3-13d4-4f46-8d06-662fa3633170","Type":"ContainerStarted","Data":"156eae2bb59b9553399c0c9d796c5fade87b2e8cea11a1f6ca8521367f29f051"}
Oct 31 01:07:52.816920 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:52.816833    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9" event={"ID":"46801ca3-13d4-4f46-8d06-662fa3633170","Type":"ContainerStarted","Data":"73f13bb1219884a3d0d78576cf451261ddf893318cb945d3c109deb2b2d3b9e9"}
Oct 31 01:07:52.936111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:52.936053    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ht5d9" podStartSLOduration=2.125713828 podStartE2EDuration="4.936039003s" totalImagesPullingTime="2.810325175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:52.93439753 +0000 UTC m=+189.097540955" watchObservedRunningTime="2026-10-31 01:07:52.936039003 +0000 UTC m=+189.099182415"
Oct 31 01:07:59.872023 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.871988    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"]
Oct 31 01:07:59.874237 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.874219    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.954333 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.954296    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 01:07:59.954514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.954374    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 01:07:59.967606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.967573    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.967606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.967604    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.967871 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.967629    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfbn\" (UniqueName: \"kubernetes.io/projected/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-kube-api-access-xkfbn\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.967871 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.967725    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.967871 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.967828    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.967871 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.967869    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:07:59.991913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.991883    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 01:07:59.991913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.991899    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 01:07:59.992148 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.991979    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9qxzz\""
Oct 31 01:07:59.992148 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.992016    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 01:07:59.993876 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:07:59.993852    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"]
Oct 31 01:08:00.069236 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069201    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.069432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069250    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.069432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069281    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.069432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069301    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfbn\" (UniqueName: \"kubernetes.io/projected/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-kube-api-access-xkfbn\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.069432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069319    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.069432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069385    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.069893 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.069862    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.072006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.071974    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.072115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.072083    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.072228 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.072210    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.072324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.072308    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.143264 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.143187    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfbn\" (UniqueName: \"kubernetes.io/projected/ee11ca63-ef7e-405e-bb4a-c44fcf8a5081-kube-api-access-xkfbn\") pod \"opendatahub-operator-controller-manager-559fd7ff56-2hltz\" (UID: \"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.186313 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.186281    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:00.716222 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:08:00.716186    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee11ca63_ef7e_405e_bb4a_c44fcf8a5081.slice/crio-4a411a585bc9bd4f7b40d76822f67f3059e82d4c36607d54d7dadbbaf539b85f WatchSource:0}: Error finding container 4a411a585bc9bd4f7b40d76822f67f3059e82d4c36607d54d7dadbbaf539b85f: Status 404 returned error can't find the container with id 4a411a585bc9bd4f7b40d76822f67f3059e82d4c36607d54d7dadbbaf539b85f
Oct 31 01:08:00.716940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.716918    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"]
Oct 31 01:08:00.851088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:00.851048    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz" event={"ID":"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081","Type":"ContainerStarted","Data":"4a411a585bc9bd4f7b40d76822f67f3059e82d4c36607d54d7dadbbaf539b85f"}
Oct 31 01:08:04.871349 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:04.871313    5718 generic.go:356] "Generic (PLEG): container finished" podID="ee11ca63-ef7e-405e-bb4a-c44fcf8a5081" containerID="c5905afb7ebdd7318c3280e04c766dfb2a431be0087351847872884d8855d85b" exitCode=0
Oct 31 01:08:04.871782 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:04.871398    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz" event={"ID":"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081","Type":"ContainerDied","Data":"c5905afb7ebdd7318c3280e04c766dfb2a431be0087351847872884d8855d85b"}
Oct 31 01:08:05.422698 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.422636    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"]
Oct 31 01:08:05.424593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.424577    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.483813 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.483775    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 01:08:05.483813 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.483785    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 01:08:05.487557 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.487526    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"]
Oct 31 01:08:05.491913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.491889    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dkcb7\""
Oct 31 01:08:05.526821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.526785    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54322f7b-53d0-4d84-be02-195f6296a5a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qb6hx\" (UID: \"54322f7b-53d0-4d84-be02-195f6296a5a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.526979 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.526926    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mh82\" (UniqueName: \"kubernetes.io/projected/54322f7b-53d0-4d84-be02-195f6296a5a7-kube-api-access-8mh82\") pod \"cert-manager-webhook-5dbbbf75cc-qb6hx\" (UID: \"54322f7b-53d0-4d84-be02-195f6296a5a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.627957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.627926    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh82\" (UniqueName: \"kubernetes.io/projected/54322f7b-53d0-4d84-be02-195f6296a5a7-kube-api-access-8mh82\") pod \"cert-manager-webhook-5dbbbf75cc-qb6hx\" (UID: \"54322f7b-53d0-4d84-be02-195f6296a5a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.628141 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.627981    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54322f7b-53d0-4d84-be02-195f6296a5a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qb6hx\" (UID: \"54322f7b-53d0-4d84-be02-195f6296a5a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.686621 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.686545    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh82\" (UniqueName: \"kubernetes.io/projected/54322f7b-53d0-4d84-be02-195f6296a5a7-kube-api-access-8mh82\") pod \"cert-manager-webhook-5dbbbf75cc-qb6hx\" (UID: \"54322f7b-53d0-4d84-be02-195f6296a5a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.687701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.687682    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54322f7b-53d0-4d84-be02-195f6296a5a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qb6hx\" (UID: \"54322f7b-53d0-4d84-be02-195f6296a5a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.734163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.734129    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:05.878869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.878819    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz" event={"ID":"ee11ca63-ef7e-405e-bb4a-c44fcf8a5081","Type":"ContainerStarted","Data":"32ee77849ba1e02b57976f1343f38eedaf9a607de2be61da169f4eb826e27883"}
Oct 31 01:08:05.879336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:05.878939    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:06.087182 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:06.087129    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz" podStartSLOduration=2.254324049 podStartE2EDuration="7.08711625s" totalImagesPullingTime="4.265578723s" totalInitContainerRuntime="567.213478ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:06.084689005 +0000 UTC m=+202.247832417" watchObservedRunningTime="2026-10-31 01:08:06.08711625 +0000 UTC m=+202.250259665"
Oct 31 01:08:06.123112 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:06.123080    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"]
Oct 31 01:08:06.138994 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:08:06.138954    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54322f7b_53d0_4d84_be02_195f6296a5a7.slice/crio-7c1da1aaf54d26ec0ace498c78363b7dbc0f125e4dbfc335ecf87326ce6293d4 WatchSource:0}: Error finding container 7c1da1aaf54d26ec0ace498c78363b7dbc0f125e4dbfc335ecf87326ce6293d4: Status 404 returned error can't find the container with id 7c1da1aaf54d26ec0ace498c78363b7dbc0f125e4dbfc335ecf87326ce6293d4
Oct 31 01:08:06.884727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:06.884681    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx" event={"ID":"54322f7b-53d0-4d84-be02-195f6296a5a7","Type":"ContainerStarted","Data":"7c1da1aaf54d26ec0ace498c78363b7dbc0f125e4dbfc335ecf87326ce6293d4"}
Oct 31 01:08:07.940589 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:08:07.940545    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:08:10.905947 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:10.905866    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx" event={"ID":"54322f7b-53d0-4d84-be02-195f6296a5a7","Type":"ContainerStarted","Data":"e2e734253daf78d8e19324b0dd706acbc5acc3e844415ca10842672f086458f1"}
Oct 31 01:08:10.906353 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:10.905985    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:11.027773 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:11.027693    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx" podStartSLOduration=1.562418321 podStartE2EDuration="6.027670421s" totalImagesPullingTime="4.4652521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:11.026975022 +0000 UTC m=+207.190118439" watchObservedRunningTime="2026-10-31 01:08:11.027670421 +0000 UTC m=+207.190813835"
Oct 31 01:08:15.316377 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.316340    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-k64tr"]
Oct 31 01:08:15.319532 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.319515    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.378108 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.378074    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-k64tr"]
Oct 31 01:08:15.419362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.419330    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407-bound-sa-token\") pod \"cert-manager-597b95cb84-k64tr\" (UID: \"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407\") " pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.419526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.419372    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfsj\" (UniqueName: \"kubernetes.io/projected/0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407-kube-api-access-4xfsj\") pod \"cert-manager-597b95cb84-k64tr\" (UID: \"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407\") " pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.424413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.424395    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xlrf8\""
Oct 31 01:08:15.520410 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.520377    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407-bound-sa-token\") pod \"cert-manager-597b95cb84-k64tr\" (UID: \"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407\") " pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.520590 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.520419    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfsj\" (UniqueName: \"kubernetes.io/projected/0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407-kube-api-access-4xfsj\") pod \"cert-manager-597b95cb84-k64tr\" (UID: \"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407\") " pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.587219 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.587144    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfsj\" (UniqueName: \"kubernetes.io/projected/0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407-kube-api-access-4xfsj\") pod \"cert-manager-597b95cb84-k64tr\" (UID: \"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407\") " pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.588957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.588936    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407-bound-sa-token\") pod \"cert-manager-597b95cb84-k64tr\" (UID: \"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407\") " pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:15.630700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:15.630672    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-k64tr"
Oct 31 01:08:16.013118 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:16.013094    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-k64tr"]
Oct 31 01:08:16.015836 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:08:16.015798    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fe95ead_d8b9_4dcb_8b53_6ca36fa3b407.slice/crio-243ae9067310b7cbc6cb80feec7d780e06ce365ccb8531784f9ca235843f0b5d WatchSource:0}: Error finding container 243ae9067310b7cbc6cb80feec7d780e06ce365ccb8531784f9ca235843f0b5d: Status 404 returned error can't find the container with id 243ae9067310b7cbc6cb80feec7d780e06ce365ccb8531784f9ca235843f0b5d
Oct 31 01:08:16.887365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:16.887328    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-2hltz"
Oct 31 01:08:16.912540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:16.912516    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qb6hx"
Oct 31 01:08:16.934622 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:16.934576    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-k64tr" event={"ID":"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407","Type":"ContainerStarted","Data":"9526515fbc858b29b55de0c17aae6a901a7e7f69d4ee5686d2dfb17093f54f4a"}
Oct 31 01:08:16.934831 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:16.934639    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-k64tr" event={"ID":"0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407","Type":"ContainerStarted","Data":"243ae9067310b7cbc6cb80feec7d780e06ce365ccb8531784f9ca235843f0b5d"}
Oct 31 01:08:17.354406 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:17.354354    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-k64tr" podStartSLOduration=2.3543409029999998 podStartE2EDuration="2.354340903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:17.353804504 +0000 UTC m=+213.516947918" watchObservedRunningTime="2026-10-31 01:08:17.354340903 +0000 UTC m=+213.517484308"
Oct 31 01:08:26.701211 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.701171    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"]
Oct 31 01:08:26.704912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.704895    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.759820 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.759791    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 01:08:26.760113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.760086    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 01:08:26.761207 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.761185    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"]
Oct 31 01:08:26.810073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.810043    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 01:08:26.810073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.810043    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 01:08:26.810258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.810105    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6qzfw\""
Oct 31 01:08:26.816397 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.816364    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fv7w\" (UniqueName: \"kubernetes.io/projected/d95ec0ac-a995-42ea-acfa-d3566d1d742b-kube-api-access-9fv7w\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.816544 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.816408    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d95ec0ac-a995-42ea-acfa-d3566d1d742b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.816544 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.816436    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95ec0ac-a995-42ea-acfa-d3566d1d742b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.917356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.917325    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fv7w\" (UniqueName: \"kubernetes.io/projected/d95ec0ac-a995-42ea-acfa-d3566d1d742b-kube-api-access-9fv7w\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.917547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.917367    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d95ec0ac-a995-42ea-acfa-d3566d1d742b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.917547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.917404    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95ec0ac-a995-42ea-acfa-d3566d1d742b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.918039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.918018    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d95ec0ac-a995-42ea-acfa-d3566d1d742b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.920149 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.920126    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95ec0ac-a995-42ea-acfa-d3566d1d742b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:26.975073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:26.974997    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fv7w\" (UniqueName: \"kubernetes.io/projected/d95ec0ac-a995-42ea-acfa-d3566d1d742b-kube-api-access-9fv7w\") pod \"kuadrant-console-plugin-66c4fb575d-m5wd8\" (UID: \"d95ec0ac-a995-42ea-acfa-d3566d1d742b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:27.017801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:27.017767    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"
Oct 31 01:08:27.399248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:27.399223    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8"]
Oct 31 01:08:27.401231 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:08:27.401200    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95ec0ac_a995_42ea_acfa_d3566d1d742b.slice/crio-b5f39b0f8a97a82f8fe472bd460d706aa026e032bae7e156d55d75c72d61b8e2 WatchSource:0}: Error finding container b5f39b0f8a97a82f8fe472bd460d706aa026e032bae7e156d55d75c72d61b8e2: Status 404 returned error can't find the container with id b5f39b0f8a97a82f8fe472bd460d706aa026e032bae7e156d55d75c72d61b8e2
Oct 31 01:08:27.981030 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:27.980996    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8" event={"ID":"d95ec0ac-a995-42ea-acfa-d3566d1d742b","Type":"ContainerStarted","Data":"b5f39b0f8a97a82f8fe472bd460d706aa026e032bae7e156d55d75c72d61b8e2"}
Oct 31 01:08:34.009839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:34.009800    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8" event={"ID":"d95ec0ac-a995-42ea-acfa-d3566d1d742b","Type":"ContainerStarted","Data":"b6a333b5a5fa0cfc55fb39c11a96720405705c58f68b55bed83467ae5d2f6468"}
Oct 31 01:08:34.128996 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:34.128945    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m5wd8" podStartSLOduration=2.065688065 podStartE2EDuration="8.128931453s" totalImagesPullingTime="6.063243388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:34.12833247 +0000 UTC m=+230.291475886" watchObservedRunningTime="2026-10-31 01:08:34.128931453 +0000 UTC m=+230.292074869"
Oct 31 01:08:37.923204 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:08:37.923158    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:08:51.832229 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.832195    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"]
Oct 31 01:08:51.839031 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.839009    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:51.892171 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.892141    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 01:08:51.892328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.892197    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xnxmt\""
Oct 31 01:08:51.942644 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.942603    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 01:08:51.943964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.943938    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"]
Oct 31 01:08:51.947220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.947198    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-tls-certs\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:51.947281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.947256    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-tmp\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:51.947359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:51.947345    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jxb\" (UniqueName: \"kubernetes.io/projected/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-kube-api-access-p2jxb\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.048567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.048525    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jxb\" (UniqueName: \"kubernetes.io/projected/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-kube-api-access-p2jxb\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.048796 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.048591    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-tls-certs\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.048796 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.048629    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-tmp\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.051085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.051058    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-tmp\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.051265 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.051247    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-tls-certs\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.106816 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.106753    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jxb\" (UniqueName: \"kubernetes.io/projected/6aba0896-e4fb-4b1d-9904-ca4e55c59e82-kube-api-access-p2jxb\") pod \"kube-auth-proxy-7ff59956fc-j96vj\" (UID: \"6aba0896-e4fb-4b1d-9904-ca4e55c59e82\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.149693 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.149639    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"
Oct 31 01:08:52.524528 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:52.524498    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj"]
Oct 31 01:08:52.526981 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:08:52.526952    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aba0896_e4fb_4b1d_9904_ca4e55c59e82.slice/crio-7998d6abf9ee3480e9d0483a7495f3d32214839b7a3043700c56f4f792e806d8 WatchSource:0}: Error finding container 7998d6abf9ee3480e9d0483a7495f3d32214839b7a3043700c56f4f792e806d8: Status 404 returned error can't find the container with id 7998d6abf9ee3480e9d0483a7495f3d32214839b7a3043700c56f4f792e806d8
Oct 31 01:08:53.094223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:53.094185    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj" event={"ID":"6aba0896-e4fb-4b1d-9904-ca4e55c59e82","Type":"ContainerStarted","Data":"7998d6abf9ee3480e9d0483a7495f3d32214839b7a3043700c56f4f792e806d8"}
Oct 31 01:08:56.109055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:56.109017    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj" event={"ID":"6aba0896-e4fb-4b1d-9904-ca4e55c59e82","Type":"ContainerStarted","Data":"66385fbaf88adb4640d917d64195bf4f650126a38db428f234b3b88b1b0427c9"}
Oct 31 01:08:56.227981 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:08:56.227925    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-j96vj" podStartSLOduration=2.276228291 podStartE2EDuration="5.227912073s" totalImagesPullingTime="2.951683782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:56.226386984 +0000 UTC m=+252.389530396" watchObservedRunningTime="2026-10-31 01:08:56.227912073 +0000 UTC m=+252.391055488"
Oct 31 01:09:02.794926 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.794892    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"]
Oct 31 01:09:02.798373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.798355    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.858249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.858211    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.858431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.858305    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.858431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.858375    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.858431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.858407    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcfkd\" (UniqueName: \"kubernetes.io/projected/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-kube-api-access-vcfkd\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.908951 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.908897    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 01:09:02.909127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.908951    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-f9xvw\""
Oct 31 01:09:02.909127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.908998    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 01:09:02.958975 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.958943    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.959143 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.958999    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.959143 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.959019    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.959143 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.959036    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcfkd\" (UniqueName: \"kubernetes.io/projected/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-kube-api-access-vcfkd\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.959320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.959303    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:02.959362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:02.959335    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:03.011748 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:03.011712    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 01:09:03.012785 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:03.012760    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"]
Oct 31 01:09:03.017183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:03.017163    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcfkd\" (UniqueName: \"kubernetes.io/projected/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-kube-api-access-vcfkd\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:03.020830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:03.020806    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eee47c4c-dd7a-43e7-a91d-9407c14c5c06-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sg6nz\" (UID: \"eee47c4c-dd7a-43e7-a91d-9407c14c5c06\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:03.111388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:03.111286    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:03.534086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:03.534051    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"]
Oct 31 01:09:03.572978 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:09:03.572942    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee47c4c_dd7a_43e7_a91d_9407c14c5c06.slice/crio-cb82e6104e3d5ad9e8e7943bc583c604b533fe85c0d08d805874ac2a13af865c WatchSource:0}: Error finding container cb82e6104e3d5ad9e8e7943bc583c604b533fe85c0d08d805874ac2a13af865c: Status 404 returned error can't find the container with id cb82e6104e3d5ad9e8e7943bc583c604b533fe85c0d08d805874ac2a13af865c
Oct 31 01:09:04.154809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:04.154770    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz" event={"ID":"eee47c4c-dd7a-43e7-a91d-9407c14c5c06","Type":"ContainerStarted","Data":"cb82e6104e3d5ad9e8e7943bc583c604b533fe85c0d08d805874ac2a13af865c"}
Oct 31 01:09:06.166173 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:06.166135    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz" event={"ID":"eee47c4c-dd7a-43e7-a91d-9407c14c5c06","Type":"ContainerStarted","Data":"d65edc62617691bfb9b8bfe59361aaf1fbc462592b01aa7eab04e85252cf1c2a"}
Oct 31 01:09:06.166589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:06.166260    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:06.336524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:06.336467    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz" podStartSLOduration=1.9916245639999999 podStartE2EDuration="4.336452732s" totalImagesPullingTime="2.344828168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:06.335297387 +0000 UTC m=+262.498440805" watchObservedRunningTime="2026-10-31 01:09:06.336452732 +0000 UTC m=+262.499596191"
Oct 31 01:09:07.925003 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:09:07.924964    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:09:07.934168 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:07.934141    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:07.937665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:07.937636    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:07.992886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:07.992857    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 01:09:07.994322 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:07.994293    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:08.010446 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.010419    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rc6\" (UniqueName: \"kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6\") pod \"limitador-limitador-74dfcfb894-wjnkz\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.010570 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.010475    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file\") pod \"limitador-limitador-74dfcfb894-wjnkz\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.113508 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.113461    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file\") pod \"limitador-limitador-74dfcfb894-wjnkz\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.113757 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.113578    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rc6\" (UniqueName: \"kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6\") pod \"limitador-limitador-74dfcfb894-wjnkz\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.114401 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.114371    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file\") pod \"limitador-limitador-74dfcfb894-wjnkz\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.171822 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.171790    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rc6\" (UniqueName: \"kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6\") pod \"limitador-limitador-74dfcfb894-wjnkz\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.175688 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.175608    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:08.250247 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.250214    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:08.627487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:08.627426    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:08.629757 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:09:08.629728    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d02244_5edc_4295_bfcf_724da4318dd2.slice/crio-f1207fdbb26a2d962e5443a781f44aa56cf9944af2f15e89c5b943f7521c42f7 WatchSource:0}: Error finding container f1207fdbb26a2d962e5443a781f44aa56cf9944af2f15e89c5b943f7521c42f7: Status 404 returned error can't find the container with id f1207fdbb26a2d962e5443a781f44aa56cf9944af2f15e89c5b943f7521c42f7
Oct 31 01:09:09.180662 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:09.180611    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz" event={"ID":"85d02244-5edc-4295-bfcf-724da4318dd2","Type":"ContainerStarted","Data":"f1207fdbb26a2d962e5443a781f44aa56cf9944af2f15e89c5b943f7521c42f7"}
Oct 31 01:09:13.200357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:13.200320    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz" event={"ID":"85d02244-5edc-4295-bfcf-724da4318dd2","Type":"ContainerStarted","Data":"ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3"}
Oct 31 01:09:13.200891 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:13.200469    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:13.317419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:13.317318    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz" podStartSLOduration=2.754041244 podStartE2EDuration="6.317298106s" totalImagesPullingTime="3.563256862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:13.315805433 +0000 UTC m=+269.478948849" watchObservedRunningTime="2026-10-31 01:09:13.317298106 +0000 UTC m=+269.480441523"
Oct 31 01:09:24.205804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:24.205724    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:27.171589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:27.171552    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sg6nz"
Oct 31 01:09:28.140836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.140801    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:28.141035 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.140997    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz" podUID="85d02244-5edc-4295-bfcf-724da4318dd2" containerName="limitador" containerID="cri-o://ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3" gracePeriod=30
Oct 31 01:09:28.685718 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.685694    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:28.804417 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.804375    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file\" (UniqueName: \"kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file\") pod \"85d02244-5edc-4295-bfcf-724da4318dd2\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") "
Oct 31 01:09:28.804742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.804491    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6\" (UniqueName: \"kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6\") pod \"85d02244-5edc-4295-bfcf-724da4318dd2\" (UID: \"85d02244-5edc-4295-bfcf-724da4318dd2\") "
Oct 31 01:09:28.804865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.804839    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file" pod "85d02244-5edc-4295-bfcf-724da4318dd2" (UID: "85d02244-5edc-4295-bfcf-724da4318dd2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:28.807028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.806999    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6" pod "85d02244-5edc-4295-bfcf-724da4318dd2" (UID: "85d02244-5edc-4295-bfcf-724da4318dd2"). InnerVolumeSpecName "kube-api-access-n2rc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:28.905931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.905893    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rc6\" (UniqueName: \"kubernetes.io/projected/85d02244-5edc-4295-bfcf-724da4318dd2-kube-api-access-n2rc6\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:28.905931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:28.905927    5718 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85d02244-5edc-4295-bfcf-724da4318dd2-config-file\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:29.274382 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.274345    5718 generic.go:356] "Generic (PLEG): container finished" podID="85d02244-5edc-4295-bfcf-724da4318dd2" containerID="ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3" exitCode=0
Oct 31 01:09:29.274559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.274396    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"
Oct 31 01:09:29.274559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.274430    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz" event={"ID":"85d02244-5edc-4295-bfcf-724da4318dd2","Type":"ContainerDied","Data":"ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3"}
Oct 31 01:09:29.274559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.274487    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wjnkz" event={"ID":"85d02244-5edc-4295-bfcf-724da4318dd2","Type":"ContainerDied","Data":"f1207fdbb26a2d962e5443a781f44aa56cf9944af2f15e89c5b943f7521c42f7"}
Oct 31 01:09:29.274559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.274511    5718 scope.go:122] "RemoveContainer" containerID="ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3"
Oct 31 01:09:29.284357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.284339    5718 scope.go:122] "RemoveContainer" containerID="ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3"
Oct 31 01:09:29.284608 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:09:29.284591    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3\": container with ID starting with ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3 not found: ID does not exist" containerID="ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3"
Oct 31 01:09:29.284716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.284618    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3"} err="failed to get container status \"ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3\": rpc error: code = NotFound desc = could not find container \"ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3\": container with ID starting with ade66ca2ac3259e74e36f6284696e5aeae57a4472aa6bc524b2bd0862f3590f3 not found: ID does not exist"
Oct 31 01:09:29.441600 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.441533    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:29.491773 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.491741    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wjnkz"]
Oct 31 01:09:29.965714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:29.965681    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d02244-5edc-4295-bfcf-724da4318dd2" path="/var/lib/kubelet/pods/85d02244-5edc-4295-bfcf-724da4318dd2/volumes"
Oct 31 01:09:33.200322 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.200292    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"]
Oct 31 01:09:33.200771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.200639    5718 cpu_manager.go:414] "containerMap: removing container" podUID="85d02244-5edc-4295-bfcf-724da4318dd2" containerName="limitador"
Oct 31 01:09:33.200771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.200666    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d02244-5edc-4295-bfcf-724da4318dd2" containerName="limitador"
Oct 31 01:09:33.200771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.200766    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d02244-5edc-4295-bfcf-724da4318dd2" containerName="limitador"
Oct 31 01:09:33.203192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.203170    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.257614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.257579    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:09:33.257839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.257643    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:09:33.309089 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.309059    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 01:09:33.309089 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.309100    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jmmz4\""
Oct 31 01:09:33.309323 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.309164    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"]
Oct 31 01:09:33.348051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.348017    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4kp\" (UniqueName: \"kubernetes.io/projected/2388db30-e28c-4c77-85d3-6f9e83e09eeb-kube-api-access-qq4kp\") pod \"postgres-7cdbf6dc44-zfzh7\" (UID: \"2388db30-e28c-4c77-85d3-6f9e83e09eeb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.348051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.348054    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2388db30-e28c-4c77-85d3-6f9e83e09eeb-data\") pod \"postgres-7cdbf6dc44-zfzh7\" (UID: \"2388db30-e28c-4c77-85d3-6f9e83e09eeb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.448526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.448487    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4kp\" (UniqueName: \"kubernetes.io/projected/2388db30-e28c-4c77-85d3-6f9e83e09eeb-kube-api-access-qq4kp\") pod \"postgres-7cdbf6dc44-zfzh7\" (UID: \"2388db30-e28c-4c77-85d3-6f9e83e09eeb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.448526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.448527    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2388db30-e28c-4c77-85d3-6f9e83e09eeb-data\") pod \"postgres-7cdbf6dc44-zfzh7\" (UID: \"2388db30-e28c-4c77-85d3-6f9e83e09eeb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.448972 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.448937    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2388db30-e28c-4c77-85d3-6f9e83e09eeb-data\") pod \"postgres-7cdbf6dc44-zfzh7\" (UID: \"2388db30-e28c-4c77-85d3-6f9e83e09eeb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.506912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.506877    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4kp\" (UniqueName: \"kubernetes.io/projected/2388db30-e28c-4c77-85d3-6f9e83e09eeb-kube-api-access-qq4kp\") pod \"postgres-7cdbf6dc44-zfzh7\" (UID: \"2388db30-e28c-4c77-85d3-6f9e83e09eeb\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.514806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.514780    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:33.883960 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:33.883939    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"]
Oct 31 01:09:33.886297 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:09:33.886254    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2388db30_e28c_4c77_85d3_6f9e83e09eeb.slice/crio-0fce0c0e816ad441b27acba024759a4855f5e0b93f44ed0befc35f194a3a8f82 WatchSource:0}: Error finding container 0fce0c0e816ad441b27acba024759a4855f5e0b93f44ed0befc35f194a3a8f82: Status 404 returned error can't find the container with id 0fce0c0e816ad441b27acba024759a4855f5e0b93f44ed0befc35f194a3a8f82
Oct 31 01:09:34.300061 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:34.300026    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7" event={"ID":"2388db30-e28c-4c77-85d3-6f9e83e09eeb","Type":"ContainerStarted","Data":"0fce0c0e816ad441b27acba024759a4855f5e0b93f44ed0befc35f194a3a8f82"}
Oct 31 01:09:37.926327 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:09:37.926285    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:09:40.331819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:40.331789    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7" event={"ID":"2388db30-e28c-4c77-85d3-6f9e83e09eeb","Type":"ContainerStarted","Data":"126a6d7fbeee4b88e5eabf329d8b8fd54da83f3cbcf6d3c596da187b17186123"}
Oct 31 01:09:40.332224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:40.331913    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:09:40.448573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:40.448507    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7" podStartSLOduration=1.084412014 podStartE2EDuration="7.448488015s" totalImagesPullingTime="6.364076001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:40.447703083 +0000 UTC m=+296.610846499" watchObservedRunningTime="2026-10-31 01:09:40.448488015 +0000 UTC m=+296.611631431"
Oct 31 01:09:46.367419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:09:46.367380    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-zfzh7"
Oct 31 01:10:03.642134 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.642099    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67964d995-7k9zg"]
Oct 31 01:10:03.652904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.652876    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.700036 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.700001    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67964d995-7k9zg"]
Oct 31 01:10:03.705908 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.705877    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 01:10:03.724431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.724397    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.724596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.724452    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxnhv\" (UniqueName: \"kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.724596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.724557    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.742966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.742937    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 01:10:03.742966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.742939    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sq5nb\""
Oct 31 01:10:03.742966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.742967    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 01:10:03.825739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.825692    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxnhv\" (UniqueName: \"kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.825739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.825743    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.825981 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.825825    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.828598 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.828568    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.828736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.828568    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.884197 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.884162    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxnhv\" (UniqueName: \"kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv\") pod \"maas-controller-67964d995-7k9zg\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") " pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:03.966036 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:03.965961    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:04.253909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.253874    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8c48b985-tz8ck"]
Oct 31 01:10:04.260831 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.260810    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.316712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.316664    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8c48b985-tz8ck"]
Oct 31 01:10:04.329679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.329624    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-webhook-cert\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.329851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.329752    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-metrics-certs\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.329851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.329809    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrcr\" (UniqueName: \"kubernetes.io/projected/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-kube-api-access-fqrcr\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.380706 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.380646    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67964d995-7k9zg"]
Oct 31 01:10:04.382971 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:10:04.382944    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc89a33e_82b9_4cd4_b790_c01279eeb07b.slice/crio-6c876e0852fc51272084d225a47cf2fb965bd7cc72c4be3f6e51c63d522a6bff WatchSource:0}: Error finding container 6c876e0852fc51272084d225a47cf2fb965bd7cc72c4be3f6e51c63d522a6bff: Status 404 returned error can't find the container with id 6c876e0852fc51272084d225a47cf2fb965bd7cc72c4be3f6e51c63d522a6bff
Oct 31 01:10:04.430318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.430283    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-webhook-cert\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.430489 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.430328    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-metrics-certs\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.430489 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.430351    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrcr\" (UniqueName: \"kubernetes.io/projected/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-kube-api-access-fqrcr\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.432933 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.432909    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-metrics-certs\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.433060 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.432937    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-webhook-cert\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.445158 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.445125    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67964d995-7k9zg" event={"ID":"bc89a33e-82b9-4cd4-b790-c01279eeb07b","Type":"ContainerStarted","Data":"6c876e0852fc51272084d225a47cf2fb965bd7cc72c4be3f6e51c63d522a6bff"}
Oct 31 01:10:04.490767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.490733    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrcr\" (UniqueName: \"kubernetes.io/projected/9b9dee7c-e950-4e01-a9a5-10bb19c6c863-kube-api-access-fqrcr\") pod \"maas-controller-c8c48b985-tz8ck\" (UID: \"9b9dee7c-e950-4e01-a9a5-10bb19c6c863\") " pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.573324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.573246    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:04.951736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:04.947020    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8c48b985-tz8ck"]
Oct 31 01:10:04.961384 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:10:04.961337    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9dee7c_e950_4e01_a9a5_10bb19c6c863.slice/crio-dab9327176ff2024010446b60abd4e0b2723b305d13033cd09b9037f2140afba WatchSource:0}: Error finding container dab9327176ff2024010446b60abd4e0b2723b305d13033cd09b9037f2140afba: Status 404 returned error can't find the container with id dab9327176ff2024010446b60abd4e0b2723b305d13033cd09b9037f2140afba
Oct 31 01:10:05.451010 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:05.450976    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8c48b985-tz8ck" event={"ID":"9b9dee7c-e950-4e01-a9a5-10bb19c6c863","Type":"ContainerStarted","Data":"dab9327176ff2024010446b60abd4e0b2723b305d13033cd09b9037f2140afba"}
Oct 31 01:10:07.923289 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:10:07.923248    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:10:09.947744 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:09.947709    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xl296"]
Oct 31 01:10:10.069319 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.069284    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.075931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.075901    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xl296"]
Oct 31 01:10:10.076113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.075950    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"]
Oct 31 01:10:10.105570 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.105531    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.117633 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.117609    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"]
Oct 31 01:10:10.170182 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.170150    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vsw6f"]
Oct 31 01:10:10.189829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.189793    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb7a537-48a8-4283-b6b1-3839f038c0da-cert\") pod \"kserve-controller-manager-776cc66d47-xl296\" (UID: \"6cb7a537-48a8-4283-b6b1-3839f038c0da\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.190003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.189931    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d4qqv\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.190063 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.190009    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hdv\" (UniqueName: \"kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv\") pod \"llmisvc-controller-manager-869d5dcc5d-d4qqv\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.190207 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.190177    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pf6\" (UniqueName: \"kubernetes.io/projected/6cb7a537-48a8-4283-b6b1-3839f038c0da-kube-api-access-w7pf6\") pod \"kserve-controller-manager-776cc66d47-xl296\" (UID: \"6cb7a537-48a8-4283-b6b1-3839f038c0da\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.194954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.194934    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.234971 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.234915    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 01:10:10.234971 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.234964    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 01:10:10.235225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.234963    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nf86z\""
Oct 31 01:10:10.235225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.234968    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kllvn\""
Oct 31 01:10:10.236784 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.236762    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vsw6f"]
Oct 31 01:10:10.287637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.287608    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 01:10:10.287637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.287608    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-g6rrx\""
Oct 31 01:10:10.291027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.290989    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb7a537-48a8-4283-b6b1-3839f038c0da-cert\") pod \"kserve-controller-manager-776cc66d47-xl296\" (UID: \"6cb7a537-48a8-4283-b6b1-3839f038c0da\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.291027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.291030    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d4qqv\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.291230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.291066    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hdv\" (UniqueName: \"kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv\") pod \"llmisvc-controller-manager-869d5dcc5d-d4qqv\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.291230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.291090    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hxrh\" (UniqueName: \"kubernetes.io/projected/07643799-bc60-49c3-b2e1-51e259af5ad6-kube-api-access-4hxrh\") pod \"model-serving-api-75bd89fdfc-vsw6f\" (UID: \"07643799-bc60-49c3-b2e1-51e259af5ad6\") " pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.291230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.291167    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pf6\" (UniqueName: \"kubernetes.io/projected/6cb7a537-48a8-4283-b6b1-3839f038c0da-kube-api-access-w7pf6\") pod \"kserve-controller-manager-776cc66d47-xl296\" (UID: \"6cb7a537-48a8-4283-b6b1-3839f038c0da\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.291230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.291229    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07643799-bc60-49c3-b2e1-51e259af5ad6-tls-certs\") pod \"model-serving-api-75bd89fdfc-vsw6f\" (UID: \"07643799-bc60-49c3-b2e1-51e259af5ad6\") " pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.294120 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.294094    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cb7a537-48a8-4283-b6b1-3839f038c0da-cert\") pod \"kserve-controller-manager-776cc66d47-xl296\" (UID: \"6cb7a537-48a8-4283-b6b1-3839f038c0da\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.294265 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.294132    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d4qqv\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.352362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.352334    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hdv\" (UniqueName: \"kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv\") pod \"llmisvc-controller-manager-869d5dcc5d-d4qqv\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.352527 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.352465    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pf6\" (UniqueName: \"kubernetes.io/projected/6cb7a537-48a8-4283-b6b1-3839f038c0da-kube-api-access-w7pf6\") pod \"kserve-controller-manager-776cc66d47-xl296\" (UID: \"6cb7a537-48a8-4283-b6b1-3839f038c0da\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.392068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.392033    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07643799-bc60-49c3-b2e1-51e259af5ad6-tls-certs\") pod \"model-serving-api-75bd89fdfc-vsw6f\" (UID: \"07643799-bc60-49c3-b2e1-51e259af5ad6\") " pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.392235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.392106    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hxrh\" (UniqueName: \"kubernetes.io/projected/07643799-bc60-49c3-b2e1-51e259af5ad6-kube-api-access-4hxrh\") pod \"model-serving-api-75bd89fdfc-vsw6f\" (UID: \"07643799-bc60-49c3-b2e1-51e259af5ad6\") " pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.395998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.395968    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07643799-bc60-49c3-b2e1-51e259af5ad6-tls-certs\") pod \"model-serving-api-75bd89fdfc-vsw6f\" (UID: \"07643799-bc60-49c3-b2e1-51e259af5ad6\") " pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.416198 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.416171    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:10.427348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.427323    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:10.450360 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.450340    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hxrh\" (UniqueName: \"kubernetes.io/projected/07643799-bc60-49c3-b2e1-51e259af5ad6-kube-api-access-4hxrh\") pod \"model-serving-api-75bd89fdfc-vsw6f\" (UID: \"07643799-bc60-49c3-b2e1-51e259af5ad6\") " pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.527607 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.526017    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:10.815639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.815612    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xl296"]
Oct 31 01:10:10.817812 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:10:10.817778    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb7a537_48a8_4283_b6b1_3839f038c0da.slice/crio-8406ad05f09354f8e146012271ba6d4c3f1a5f12742eaf9ac20eab5f2faca541 WatchSource:0}: Error finding container 8406ad05f09354f8e146012271ba6d4c3f1a5f12742eaf9ac20eab5f2faca541: Status 404 returned error can't find the container with id 8406ad05f09354f8e146012271ba6d4c3f1a5f12742eaf9ac20eab5f2faca541
Oct 31 01:10:10.865222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.865199    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"]
Oct 31 01:10:10.867241 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:10:10.867210    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28b030bd_f46e_49d4_b8dd_9d80ef0a0237.slice/crio-206962d632a1ed6ed37da86d5c1af0bd8d7069ff841875d7d7c379af7656ff97 WatchSource:0}: Error finding container 206962d632a1ed6ed37da86d5c1af0bd8d7069ff841875d7d7c379af7656ff97: Status 404 returned error can't find the container with id 206962d632a1ed6ed37da86d5c1af0bd8d7069ff841875d7d7c379af7656ff97
Oct 31 01:10:10.932812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:10.932636    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vsw6f"]
Oct 31 01:10:11.488641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:11.488611    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f" event={"ID":"07643799-bc60-49c3-b2e1-51e259af5ad6","Type":"ContainerStarted","Data":"8a537ac43aaf791281434ac3ee99cfcef893457fec53e48da3218c9674b2c0b8"}
Oct 31 01:10:11.489961 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:11.489937    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296" event={"ID":"6cb7a537-48a8-4283-b6b1-3839f038c0da","Type":"ContainerStarted","Data":"8406ad05f09354f8e146012271ba6d4c3f1a5f12742eaf9ac20eab5f2faca541"}
Oct 31 01:10:11.491169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:11.491146    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv" event={"ID":"28b030bd-f46e-49d4-b8dd-9d80ef0a0237","Type":"ContainerStarted","Data":"206962d632a1ed6ed37da86d5c1af0bd8d7069ff841875d7d7c379af7656ff97"}
Oct 31 01:10:12.500079 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:12.500033    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67964d995-7k9zg" event={"ID":"bc89a33e-82b9-4cd4-b790-c01279eeb07b","Type":"ContainerStarted","Data":"e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95"}
Oct 31 01:10:12.501106 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:12.501082    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:12.504297 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:12.504264    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8c48b985-tz8ck" event={"ID":"9b9dee7c-e950-4e01-a9a5-10bb19c6c863","Type":"ContainerStarted","Data":"2c4342ee2261e4773d75ca927ebf2f306f9b7151b9e0c9c82253e30dcb08159d"}
Oct 31 01:10:12.504897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:12.504857    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:12.750056 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:12.749992    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67964d995-7k9zg" podStartSLOduration=2.710179829 podStartE2EDuration="9.749972763s" totalImagesPullingTime="7.039792934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:12.622111322 +0000 UTC m=+328.785254745" watchObservedRunningTime="2026-10-31 01:10:12.749972763 +0000 UTC m=+328.913116179"
Oct 31 01:10:14.520236 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.520199    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f" event={"ID":"07643799-bc60-49c3-b2e1-51e259af5ad6","Type":"ContainerStarted","Data":"e37a7cb882b9aec096222df83823db4ec02069de6984426c16e7640f7af8db5f"}
Oct 31 01:10:14.520721 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.520349    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:14.521746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.521721    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296" event={"ID":"6cb7a537-48a8-4283-b6b1-3839f038c0da","Type":"ContainerStarted","Data":"35ef10bedfc69eb2b93e9b29a3cc20a348401327a28a3b33970c481c9a76fb7c"}
Oct 31 01:10:14.521881 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.521788    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:10:14.523211 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.523189    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv" event={"ID":"28b030bd-f46e-49d4-b8dd-9d80ef0a0237","Type":"ContainerStarted","Data":"d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f"}
Oct 31 01:10:14.523317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.523297    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:14.636492 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.636434    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-c8c48b985-tz8ck" podStartSLOduration=4.146333866 podStartE2EDuration="10.636419644s" totalImagesPullingTime="6.490085778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:12.751571423 +0000 UTC m=+328.914714838" watchObservedRunningTime="2026-10-31 01:10:14.636419644 +0000 UTC m=+330.799563051"
Oct 31 01:10:14.636715 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.636553    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f" podStartSLOduration=1.990182825 podStartE2EDuration="4.636549244s" totalImagesPullingTime="2.646366419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:14.635765954 +0000 UTC m=+330.798909371" watchObservedRunningTime="2026-10-31 01:10:14.636549244 +0000 UTC m=+330.799692688"
Oct 31 01:10:14.755102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.755038    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv" podStartSLOduration=1.483677597 podStartE2EDuration="4.755025336s" totalImagesPullingTime="3.271347739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:14.753990439 +0000 UTC m=+330.917133866" watchObservedRunningTime="2026-10-31 01:10:14.755025336 +0000 UTC m=+330.918168751"
Oct 31 01:10:14.870606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:14.870482    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296" podStartSLOduration=2.903286562 podStartE2EDuration="5.87046924s" totalImagesPullingTime="2.967182678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:14.86916457 +0000 UTC m=+331.032307986" watchObservedRunningTime="2026-10-31 01:10:14.87046924 +0000 UTC m=+331.033612654"
Oct 31 01:10:15.350262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.350227    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"]
Oct 31 01:10:15.360786 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.360744    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.411779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.411746    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"]
Oct 31 01:10:15.451425 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.451393    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.451594 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.451455    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.451637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.451601    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49dl\" (UniqueName: \"kubernetes.io/projected/2b0d8892-27d3-483d-8666-09db3fda9da8-kube-api-access-p49dl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.451698 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.451678    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b0d8892-27d3-483d-8666-09db3fda9da8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.460330 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.460297    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 01:10:15.460476 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.460375    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 01:10:15.460476 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.460390    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 01:10:15.460556 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.460485    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-ctws5\""
Oct 31 01:10:15.552547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.552520    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49dl\" (UniqueName: \"kubernetes.io/projected/2b0d8892-27d3-483d-8666-09db3fda9da8-kube-api-access-p49dl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.553071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.552563    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b0d8892-27d3-483d-8666-09db3fda9da8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.553071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.552595    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.553071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.552627    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.553071 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:10:15.552770    5718 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 31 01:10:15.553071 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:10:15.552855    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-proxy-source podName:2b0d8892-27d3-483d-8666-09db3fda9da8 nodeName:}" failed. No retries permitted until 2026-10-31 01:10:16.052831027 +0000 UTC m=+332.215974439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-lzwxw" (UID: "2b0d8892-27d3-483d-8666-09db3fda9da8") : configmap "usage-tenancy-proxy-source" not found
Oct 31 01:10:15.553369 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.553294    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.555307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.555286    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b0d8892-27d3-483d-8666-09db3fda9da8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:15.613244 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:15.613168    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49dl\" (UniqueName: \"kubernetes.io/projected/2b0d8892-27d3-483d-8666-09db3fda9da8-kube-api-access-p49dl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:16.058125 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:16.058085    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:16.058934 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:16.058913    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2b0d8892-27d3-483d-8666-09db3fda9da8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-lzwxw\" (UID: \"2b0d8892-27d3-483d-8666-09db3fda9da8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:16.273822 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:16.273784    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:16.648124 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:16.648102    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"]
Oct 31 01:10:16.650263 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:10:16.650230    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0d8892_27d3_483d_8666_09db3fda9da8.slice/crio-08c7e9872fddac4a28afc2fbe85f2a45fb9ee51fe1912703128ece7d730323f7 WatchSource:0}: Error finding container 08c7e9872fddac4a28afc2fbe85f2a45fb9ee51fe1912703128ece7d730323f7: Status 404 returned error can't find the container with id 08c7e9872fddac4a28afc2fbe85f2a45fb9ee51fe1912703128ece7d730323f7
Oct 31 01:10:17.539963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:17.539927    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw" event={"ID":"2b0d8892-27d3-483d-8666-09db3fda9da8","Type":"ContainerStarted","Data":"08c7e9872fddac4a28afc2fbe85f2a45fb9ee51fe1912703128ece7d730323f7"}
Oct 31 01:10:24.526221 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:24.526191    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8c48b985-tz8ck"
Oct 31 01:10:24.526696 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:24.526264    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:24.791011 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:24.790921    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67964d995-7k9zg"]
Oct 31 01:10:24.791180 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:24.791139    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-67964d995-7k9zg" podUID="bc89a33e-82b9-4cd4-b790-c01279eeb07b" containerName="manager" containerID="cri-o://e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95" gracePeriod=30
Oct 31 01:10:25.060987 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.060954    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:25.146388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.146349    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert\") pod \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") "
Oct 31 01:10:25.146570 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.146462    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv\" (UniqueName: \"kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv\") pod \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") "
Oct 31 01:10:25.146570 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.146542    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs\") pod \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\" (UID: \"bc89a33e-82b9-4cd4-b790-c01279eeb07b\") "
Oct 31 01:10:25.149190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.149160    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs" pod "bc89a33e-82b9-4cd4-b790-c01279eeb07b" (UID: "bc89a33e-82b9-4cd4-b790-c01279eeb07b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:25.149337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.149264    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert" pod "bc89a33e-82b9-4cd4-b790-c01279eeb07b" (UID: "bc89a33e-82b9-4cd4-b790-c01279eeb07b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:25.149505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.149476    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv" pod "bc89a33e-82b9-4cd4-b790-c01279eeb07b" (UID: "bc89a33e-82b9-4cd4-b790-c01279eeb07b"). InnerVolumeSpecName "kube-api-access-gxnhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:25.248183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.248147    5718 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-webhook-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:25.248183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.248180    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxnhv\" (UniqueName: \"kubernetes.io/projected/bc89a33e-82b9-4cd4-b790-c01279eeb07b-kube-api-access-gxnhv\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:25.248183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.248190    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc89a33e-82b9-4cd4-b790-c01279eeb07b-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:25.533443 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.533413    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vsw6f"
Oct 31 01:10:25.586362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.586328    5718 generic.go:356] "Generic (PLEG): container finished" podID="bc89a33e-82b9-4cd4-b790-c01279eeb07b" containerID="e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95" exitCode=0
Oct 31 01:10:25.586569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.586392    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67964d995-7k9zg" event={"ID":"bc89a33e-82b9-4cd4-b790-c01279eeb07b","Type":"ContainerDied","Data":"e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95"}
Oct 31 01:10:25.586569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.586426    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67964d995-7k9zg"
Oct 31 01:10:25.586569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.586447    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67964d995-7k9zg" event={"ID":"bc89a33e-82b9-4cd4-b790-c01279eeb07b","Type":"ContainerDied","Data":"6c876e0852fc51272084d225a47cf2fb965bd7cc72c4be3f6e51c63d522a6bff"}
Oct 31 01:10:25.586569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.586470    5718 scope.go:122] "RemoveContainer" containerID="e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95"
Oct 31 01:10:25.598271 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.598250    5718 scope.go:122] "RemoveContainer" containerID="e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95"
Oct 31 01:10:25.598589 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:10:25.598567    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95\": container with ID starting with e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95 not found: ID does not exist" containerID="e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95"
Oct 31 01:10:25.598703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.598602    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95"} err="failed to get container status \"e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95\": rpc error: code = NotFound desc = could not find container \"e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95\": container with ID starting with e486a24984841a249f2a8b75eca06769e76129ce99fbc6241a97181c7cb41c95 not found: ID does not exist"
Oct 31 01:10:25.765295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.765261    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67964d995-7k9zg"]
Oct 31 01:10:25.767079 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.767054    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67964d995-7k9zg"]
Oct 31 01:10:25.968060 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:25.967969    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc89a33e-82b9-4cd4-b790-c01279eeb07b" path="/var/lib/kubelet/pods/bc89a33e-82b9-4cd4-b790-c01279eeb07b/volumes"
Oct 31 01:10:30.616732 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:30.616691    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw" event={"ID":"2b0d8892-27d3-483d-8666-09db3fda9da8","Type":"ContainerStarted","Data":"b676a278947f84a071ced030b1267ce1218b654d770962ebf53007ce44596ad3"}
Oct 31 01:10:30.617206 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:30.616816    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:30.735586 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:30.735523    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw" podStartSLOduration=2.479564387 podStartE2EDuration="15.735508532s" totalImagesPullingTime="13.255944145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:30.734040052 +0000 UTC m=+346.897183472" watchObservedRunningTime="2026-10-31 01:10:30.735508532 +0000 UTC m=+346.898651946"
Oct 31 01:10:37.922937 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:10:37.922894    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:10:41.627163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:41.627132    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-lzwxw"
Oct 31 01:10:45.530343 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:45.530310    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:10:45.533369 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:10:45.533343    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xl296"
Oct 31 01:11:03.914945 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:03.914862    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"]
Oct 31 01:11:03.915325 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:03.915253    5718 cpu_manager.go:414] "containerMap: removing container" podUID="bc89a33e-82b9-4cd4-b790-c01279eeb07b" containerName="manager"
Oct 31 01:11:03.915325 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:03.915263    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc89a33e-82b9-4cd4-b790-c01279eeb07b" containerName="manager"
Oct 31 01:11:03.915399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:03.915337    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc89a33e-82b9-4cd4-b790-c01279eeb07b" containerName="manager"
Oct 31 01:11:03.926620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:03.926587    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:03.979226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:03.979196    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"]
Oct 31 01:11:04.014633 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.014600    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwmc\" (UniqueName: \"kubernetes.io/projected/1a902ea0-8131-43d8-a900-96da008d21e2-kube-api-access-vzwmc\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.014845 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.014725    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a902ea0-8131-43d8-a900-96da008d21e2-openshift-service-ca\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.014845 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.014763    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a902ea0-8131-43d8-a900-96da008d21e2-metrics-certs\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.014926 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.014902    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a902ea0-8131-43d8-a900-96da008d21e2-maas-api-tls\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.027924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.027894    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 01:11:04.030033 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.030003    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x26rq\""
Oct 31 01:11:04.030033 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.030024    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 01:11:04.115696 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.115645    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a902ea0-8131-43d8-a900-96da008d21e2-openshift-service-ca\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.115888 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.115732    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a902ea0-8131-43d8-a900-96da008d21e2-metrics-certs\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.115888 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.115834    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a902ea0-8131-43d8-a900-96da008d21e2-maas-api-tls\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.116019 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.116004    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwmc\" (UniqueName: \"kubernetes.io/projected/1a902ea0-8131-43d8-a900-96da008d21e2-kube-api-access-vzwmc\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.116437 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.116408    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a902ea0-8131-43d8-a900-96da008d21e2-openshift-service-ca\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.118966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.118936    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a902ea0-8131-43d8-a900-96da008d21e2-maas-api-tls\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.119111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.119046    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a902ea0-8131-43d8-a900-96da008d21e2-metrics-certs\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.175192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.175118    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwmc\" (UniqueName: \"kubernetes.io/projected/1a902ea0-8131-43d8-a900-96da008d21e2-kube-api-access-vzwmc\") pod \"maas-api-84cdb66567-djg7q\" (UID: \"1a902ea0-8131-43d8-a900-96da008d21e2\") " pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.238578 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.238545    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:04.611607 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.611576    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"]
Oct 31 01:11:04.613482 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:11:04.613455    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a902ea0_8131_43d8_a900_96da008d21e2.slice/crio-e8f79cd77362d0866d567054ac930624c9275984007d3a2a99bd1188d597b063 WatchSource:0}: Error finding container e8f79cd77362d0866d567054ac930624c9275984007d3a2a99bd1188d597b063: Status 404 returned error can't find the container with id e8f79cd77362d0866d567054ac930624c9275984007d3a2a99bd1188d597b063
Oct 31 01:11:04.781444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:04.781408    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q" event={"ID":"1a902ea0-8131-43d8-a900-96da008d21e2","Type":"ContainerStarted","Data":"e8f79cd77362d0866d567054ac930624c9275984007d3a2a99bd1188d597b063"}
Oct 31 01:11:06.794025 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:06.793986    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q" event={"ID":"1a902ea0-8131-43d8-a900-96da008d21e2","Type":"ContainerStarted","Data":"c02605aa1bf7e7685e5732749a2fc0605d691e04604829b55fca37490d53bdd1"}
Oct 31 01:11:06.794495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:06.794135    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:06.913243 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:06.913139    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q" podStartSLOduration=1.92541059 podStartE2EDuration="3.913123561s" totalImagesPullingTime="1.987712971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:06.912088757 +0000 UTC m=+383.075232185" watchObservedRunningTime="2026-10-31 01:11:06.913123561 +0000 UTC m=+383.076266975"
Oct 31 01:11:07.923425 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:11:07.923383    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:11:12.804891 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:12.804863    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84cdb66567-djg7q"
Oct 31 01:11:25.129520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.129479    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"]
Oct 31 01:11:25.132887 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.132866    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.191441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.191403    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"]
Oct 31 01:11:25.216903 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.216873    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e00610c-0084-4ffe-a354-7d444869288f-cert\") pod \"llmisvc-controller-manager-6466c554d8-dcfg5\" (UID: \"9e00610c-0084-4ffe-a354-7d444869288f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.217089 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.216945    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqs7\" (UniqueName: \"kubernetes.io/projected/9e00610c-0084-4ffe-a354-7d444869288f-kube-api-access-lrqs7\") pod \"llmisvc-controller-manager-6466c554d8-dcfg5\" (UID: \"9e00610c-0084-4ffe-a354-7d444869288f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.318337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.318293    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqs7\" (UniqueName: \"kubernetes.io/projected/9e00610c-0084-4ffe-a354-7d444869288f-kube-api-access-lrqs7\") pod \"llmisvc-controller-manager-6466c554d8-dcfg5\" (UID: \"9e00610c-0084-4ffe-a354-7d444869288f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.318549 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.318377    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e00610c-0084-4ffe-a354-7d444869288f-cert\") pod \"llmisvc-controller-manager-6466c554d8-dcfg5\" (UID: \"9e00610c-0084-4ffe-a354-7d444869288f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.321009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.320981    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e00610c-0084-4ffe-a354-7d444869288f-cert\") pod \"llmisvc-controller-manager-6466c554d8-dcfg5\" (UID: \"9e00610c-0084-4ffe-a354-7d444869288f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.376709 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.376638    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqs7\" (UniqueName: \"kubernetes.io/projected/9e00610c-0084-4ffe-a354-7d444869288f-kube-api-access-lrqs7\") pod \"llmisvc-controller-manager-6466c554d8-dcfg5\" (UID: \"9e00610c-0084-4ffe-a354-7d444869288f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.445603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.445517    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:25.816348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.816316    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"]
Oct 31 01:11:25.818537 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:11:25.818502    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e00610c_0084_4ffe_a354_7d444869288f.slice/crio-8c89b9b3fabb4b89a7f27875fed52ef21e8ce5b86f6aef6fda39b013e5052aa0 WatchSource:0}: Error finding container 8c89b9b3fabb4b89a7f27875fed52ef21e8ce5b86f6aef6fda39b013e5052aa0: Status 404 returned error can't find the container with id 8c89b9b3fabb4b89a7f27875fed52ef21e8ce5b86f6aef6fda39b013e5052aa0
Oct 31 01:11:25.881472 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:25.881434    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5" event={"ID":"9e00610c-0084-4ffe-a354-7d444869288f","Type":"ContainerStarted","Data":"8c89b9b3fabb4b89a7f27875fed52ef21e8ce5b86f6aef6fda39b013e5052aa0"}
Oct 31 01:11:26.888837 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:26.888742    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5" event={"ID":"9e00610c-0084-4ffe-a354-7d444869288f","Type":"ContainerStarted","Data":"d68418e35967acf3d0cb28f9ba66cc1b29449136c1d087208f64134d611c8935"}
Oct 31 01:11:26.889197 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:26.888833    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:27.007902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:27.007848    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5" podStartSLOduration=1.30004334 podStartE2EDuration="2.007832594s" totalImagesPullingTime="707.789254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:27.006635564 +0000 UTC m=+403.169778979" watchObservedRunningTime="2026-10-31 01:11:27.007832594 +0000 UTC m=+403.170976005"
Oct 31 01:11:37.923520 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:11:37.923484    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:11:57.895832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:57.895800    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dcfg5"
Oct 31 01:11:58.064527 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.064487    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"]
Oct 31 01:11:58.064754 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.064728    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv" podUID="28b030bd-f46e-49d4-b8dd-9d80ef0a0237" containerName="manager" containerID="cri-o://d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f" gracePeriod=30
Oct 31 01:11:58.317069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.317044    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:11:58.421363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.421331    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv\" (UniqueName: \"kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv\") pod \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") "
Oct 31 01:11:58.421540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.421379    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert\" (UniqueName: \"kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert\") pod \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\" (UID: \"28b030bd-f46e-49d4-b8dd-9d80ef0a0237\") "
Oct 31 01:11:58.423495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.423462    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert" pod "28b030bd-f46e-49d4-b8dd-9d80ef0a0237" (UID: "28b030bd-f46e-49d4-b8dd-9d80ef0a0237"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:58.423623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.423551    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv" pod "28b030bd-f46e-49d4-b8dd-9d80ef0a0237" (UID: "28b030bd-f46e-49d4-b8dd-9d80ef0a0237"). InnerVolumeSpecName "kube-api-access-r2hdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:58.523102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.523066    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2hdv\" (UniqueName: \"kubernetes.io/projected/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-kube-api-access-r2hdv\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:58.523102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:58.523093    5718 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28b030bd-f46e-49d4-b8dd-9d80ef0a0237-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:59.038825 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.038787    5718 generic.go:356] "Generic (PLEG): container finished" podID="28b030bd-f46e-49d4-b8dd-9d80ef0a0237" containerID="d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f" exitCode=0
Oct 31 01:11:59.039224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.038838    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"
Oct 31 01:11:59.039224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.038868    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv" event={"ID":"28b030bd-f46e-49d4-b8dd-9d80ef0a0237","Type":"ContainerDied","Data":"d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f"}
Oct 31 01:11:59.039224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.038919    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv" event={"ID":"28b030bd-f46e-49d4-b8dd-9d80ef0a0237","Type":"ContainerDied","Data":"206962d632a1ed6ed37da86d5c1af0bd8d7069ff841875d7d7c379af7656ff97"}
Oct 31 01:11:59.039224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.038941    5718 scope.go:122] "RemoveContainer" containerID="d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f"
Oct 31 01:11:59.048164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.048144    5718 scope.go:122] "RemoveContainer" containerID="d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f"
Oct 31 01:11:59.048462 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:11:59.048442    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f\": container with ID starting with d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f not found: ID does not exist" containerID="d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f"
Oct 31 01:11:59.048527 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.048476    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f"} err="failed to get container status \"d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f\": rpc error: code = NotFound desc = could not find container \"d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f\": container with ID starting with d40aabdb0ef73049e0c47818fad15f6c2ba58e62b28421ded6b535cb2990595f not found: ID does not exist"
Oct 31 01:11:59.206487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.206450    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"]
Oct 31 01:11:59.210845 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.210816    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4qqv"]
Oct 31 01:11:59.965890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:11:59.965852    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b030bd-f46e-49d4-b8dd-9d80ef0a0237" path="/var/lib/kubelet/pods/28b030bd-f46e-49d4-b8dd-9d80ef0a0237/volumes"
Oct 31 01:12:07.924193 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:12:07.924154    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:12:37.923768 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:12:37.923731    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:13:07.923367 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:13:07.923316    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:13:37.924191 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:13:37.924147    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:13:56.354864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.354779    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"]
Oct 31 01:13:56.355337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.355306    5718 cpu_manager.go:414] "containerMap: removing container" podUID="28b030bd-f46e-49d4-b8dd-9d80ef0a0237" containerName="manager"
Oct 31 01:13:56.355337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.355322    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b030bd-f46e-49d4-b8dd-9d80ef0a0237" containerName="manager"
Oct 31 01:13:56.355448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.355393    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b030bd-f46e-49d4-b8dd-9d80ef0a0237" containerName="manager"
Oct 31 01:13:56.362610 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.362579    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.415786 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.415755    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bnzrd\""
Oct 31 01:13:56.438164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438114    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438177    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9kh\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438202    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438309    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438478 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438402    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438478 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438425    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438478 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438443    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438509    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.438579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.438555    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.468582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.468551    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"]
Oct 31 01:13:56.539955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.539919    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.539966    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540105    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540153    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540185    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540235    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540295    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9kh\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540339    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540372    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540386    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540405    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540734 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540622    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540786    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.540938 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.540919    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.542790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.542761    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.542994 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.542976    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.601932 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.601893    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9kh\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.602081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.601894    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-44qpc\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:56.678085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:56.677987    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:57.101113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.101074    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"]
Oct 31 01:13:57.103156 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:13:57.103126    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a682ba_22d6_45a4_a9b9_713e5c59447a.slice/crio-6685c4417d47bd36ebc1dfa1c289fc0e6236f01424ec58073356f9d90fe31b75 WatchSource:0}: Error finding container 6685c4417d47bd36ebc1dfa1c289fc0e6236f01424ec58073356f9d90fe31b75: Status 404 returned error can't find the container with id 6685c4417d47bd36ebc1dfa1c289fc0e6236f01424ec58073356f9d90fe31b75
Oct 31 01:13:57.105463 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.105422    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:13:57.105541 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.105493    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:13:57.105541 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.105524    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:13:57.597300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.597258    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" event={"ID":"74a682ba-22d6-45a4-a9b9-713e5c59447a","Type":"ContainerStarted","Data":"8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f"}
Oct 31 01:13:57.597754 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.597318    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" event={"ID":"74a682ba-22d6-45a4-a9b9-713e5c59447a","Type":"ContainerStarted","Data":"6685c4417d47bd36ebc1dfa1c289fc0e6236f01424ec58073356f9d90fe31b75"}
Oct 31 01:13:57.678509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.678456    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:13:57.727372 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:57.727306    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podStartSLOduration=1.72728721 podStartE2EDuration="1.72728721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:57.726042309 +0000 UTC m=+553.889185725" watchObservedRunningTime="2026-10-31 01:13:57.72728721 +0000 UTC m=+553.890430626"
Oct 31 01:13:58.680368 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:58.680330    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-44qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:13:58.680789 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:58.680387    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:13:59.679110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:59.679070    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-44qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:13:59.679442 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:13:59.679149    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:14:00.679270 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:00.679230    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-44qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:14:00.679711 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:00.679291    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:14:01.678583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:01.678544    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-44qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:14:01.678804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:01.678610    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:14:02.678820 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:02.678779    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-44qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:14:02.679231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:02.678843    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:14:03.678419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:03.678381    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-44qpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 01:14:03.678597 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:03.678440    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 01:14:05.343265 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:05.343231    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:14:05.343964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:05.343919    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:14:05.344681 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:14:05.344637    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:14:07.923036 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:14:07.922985    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:14:37.922798 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:14:37.922760    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:15:00.172085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.172047    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"]
Oct 31 01:15:00.175807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.175788    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.258458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.258420    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.258642 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.258570    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcw7\" (UniqueName: \"kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7\") pod \"maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.270363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.270324    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"]
Oct 31 01:15:00.291107 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.291066    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:15:00.359174 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.359131    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcw7\" (UniqueName: \"kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7\") pod \"maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.359355 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.359207    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.359876 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.359856    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.418419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.418390    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcw7\" (UniqueName: \"kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7\") pod \"maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.489368 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.489280    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:00.882814 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:15:00.882782    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb56f72_6516_4e54_ac75_b811e416e030.slice/crio-4d1e80d588c3fddb64ef88cdd310b181a82050a1444add4a8dcb05df51807d49 WatchSource:0}: Error finding container 4d1e80d588c3fddb64ef88cdd310b181a82050a1444add4a8dcb05df51807d49: Status 404 returned error can't find the container with id 4d1e80d588c3fddb64ef88cdd310b181a82050a1444add4a8dcb05df51807d49
Oct 31 01:15:00.882928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.882822    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"]
Oct 31 01:15:00.884817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.884791    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:15:00.909508 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:00.909471    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" event={"ID":"8fb56f72-6516-4e54-ac75-b811e416e030","Type":"ContainerStarted","Data":"4d1e80d588c3fddb64ef88cdd310b181a82050a1444add4a8dcb05df51807d49"}
Oct 31 01:15:04.929939 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:04.929904    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" event={"ID":"8fb56f72-6516-4e54-ac75-b811e416e030","Type":"ContainerStarted","Data":"1f0e1f7e56a0ed743fe6094e56b0db210dd4cf43d22fcf6bff069425d00e36da"}
Oct 31 01:15:05.047240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:05.047125    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" podStartSLOduration=1.480513702 podStartE2EDuration="5.047105204s" totalImagesPullingTime="3.566591502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:05.045893342 +0000 UTC m=+621.209036757" watchObservedRunningTime="2026-10-31 01:15:05.047105204 +0000 UTC m=+621.210248633"
Oct 31 01:15:05.942245 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:05.942204    5718 generic.go:356] "Generic (PLEG): container finished" podID="8fb56f72-6516-4e54-ac75-b811e416e030" containerID="1f0e1f7e56a0ed743fe6094e56b0db210dd4cf43d22fcf6bff069425d00e36da" exitCode=7
Oct 31 01:15:05.942797 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:05.942762    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" event={"ID":"8fb56f72-6516-4e54-ac75-b811e416e030","Type":"ContainerDied","Data":"1f0e1f7e56a0ed743fe6094e56b0db210dd4cf43d22fcf6bff069425d00e36da"}
Oct 31 01:15:05.943188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:05.943162    5718 scope.go:122] "RemoveContainer" containerID="1f0e1f7e56a0ed743fe6094e56b0db210dd4cf43d22fcf6bff069425d00e36da"
Oct 31 01:15:06.950504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:06.950463    5718 generic.go:356] "Generic (PLEG): container finished" podID="8fb56f72-6516-4e54-ac75-b811e416e030" containerID="e8f3ea556052da6dae1e451319fb89f3a3f522a3bf23c07c8a2c4f0e500a0bd6" exitCode=60
Oct 31 01:15:06.950988 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:06.950550    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" event={"ID":"8fb56f72-6516-4e54-ac75-b811e416e030","Type":"ContainerDied","Data":"e8f3ea556052da6dae1e451319fb89f3a3f522a3bf23c07c8a2c4f0e500a0bd6"}
Oct 31 01:15:06.950988 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:06.950618    5718 scope.go:122] "RemoveContainer" containerID="1f0e1f7e56a0ed743fe6094e56b0db210dd4cf43d22fcf6bff069425d00e36da"
Oct 31 01:15:06.950988 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:06.950951    5718 scope.go:122] "RemoveContainer" containerID="e8f3ea556052da6dae1e451319fb89f3a3f522a3bf23c07c8a2c4f0e500a0bd6"
Oct 31 01:15:06.951266 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:06.951233    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn_odh-ai-gateway-infra(8fb56f72-6516-4e54-ac75-b811e416e030)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" podUID="8fb56f72-6516-4e54-ac75-b811e416e030"
Oct 31 01:15:07.923586 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:07.923548    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:15:07.959140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:07.959091    5718 scope.go:122] "RemoveContainer" containerID="e8f3ea556052da6dae1e451319fb89f3a3f522a3bf23c07c8a2c4f0e500a0bd6"
Oct 31 01:15:07.959642 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:07.959363    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn_odh-ai-gateway-infra(8fb56f72-6516-4e54-ac75-b811e416e030)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" podUID="8fb56f72-6516-4e54-ac75-b811e416e030"
Oct 31 01:15:11.592583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.592543    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"]
Oct 31 01:15:11.597540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.597517    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.651780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.651750    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b9656f65-serving-cert\""
Oct 31 01:15:11.666105 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.666067    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.666266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.666131    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.666266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.666226    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4xj\" (UniqueName: \"kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.666380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.666360    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.703733 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.703700    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"]
Oct 31 01:15:11.767042 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.767013    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4xj\" (UniqueName: \"kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.767227 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.767078    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.767227 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.767120    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.767227 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.767146    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.767411 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:11.767262    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-serving-cert: secret "maas-api-e2e-derive-b9656f65-serving-cert" not found
Oct 31 01:15:11.767411 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:11.767364    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls podName:6165b123-6b3c-4fdb-9729-ad19c78cc62d nodeName:}" failed. No retries permitted until 2026-10-31 01:15:12.26733033 +0000 UTC m=+628.430473740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls") pod "maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" (UID: "6165b123-6b3c-4fdb-9729-ad19c78cc62d") : secret "maas-api-e2e-derive-b9656f65-serving-cert" not found
Oct 31 01:15:11.768177 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.768157    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.769745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.769721    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:11.827568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:11.827538    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4xj\" (UniqueName: \"kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:12.273865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:12.273818    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:12.276462 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:12.276436    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\") pod \"maas-api-e2e-derive-b9656f65-79c688f996-nz5bc\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:12.511093 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:12.511058    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:12.886550 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:12.886522    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"]
Oct 31 01:15:12.889126 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:15:12.889089    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6165b123_6b3c_4fdb_9729_ad19c78cc62d.slice/crio-52f9cb0d44f388864253fe32d8d61e9d640128e6d167c8eea68c91b3c30bb979 WatchSource:0}: Error finding container 52f9cb0d44f388864253fe32d8d61e9d640128e6d167c8eea68c91b3c30bb979: Status 404 returned error can't find the container with id 52f9cb0d44f388864253fe32d8d61e9d640128e6d167c8eea68c91b3c30bb979
Oct 31 01:15:12.984578 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:12.984541    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" event={"ID":"6165b123-6b3c-4fdb-9729-ad19c78cc62d","Type":"ContainerStarted","Data":"52f9cb0d44f388864253fe32d8d61e9d640128e6d167c8eea68c91b3c30bb979"}
Oct 31 01:15:13.990402 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:13.990306    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" event={"ID":"6165b123-6b3c-4fdb-9729-ad19c78cc62d","Type":"ContainerStarted","Data":"3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95"}
Oct 31 01:15:13.990402 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:13.990389    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:14.110423 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:14.110364    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" podStartSLOduration=2.267263757 podStartE2EDuration="3.110348525s" totalImagesPullingTime="843.084768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:14.109350901 +0000 UTC m=+630.272494318" watchObservedRunningTime="2026-10-31 01:15:14.110348525 +0000 UTC m=+630.273491940"
Oct 31 01:15:20.001085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:20.001054    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:15:20.960593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:20.960561    5718 scope.go:122] "RemoveContainer" containerID="e8f3ea556052da6dae1e451319fb89f3a3f522a3bf23c07c8a2c4f0e500a0bd6"
Oct 31 01:15:22.034839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:22.034804    5718 generic.go:356] "Generic (PLEG): container finished" podID="8fb56f72-6516-4e54-ac75-b811e416e030" containerID="cfd96c187417db12fd2ca7e8ef90625d3caf4645cc55fa14be1250fc831bc4ea" exitCode=60
Oct 31 01:15:22.035240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:22.034882    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" event={"ID":"8fb56f72-6516-4e54-ac75-b811e416e030","Type":"ContainerDied","Data":"cfd96c187417db12fd2ca7e8ef90625d3caf4645cc55fa14be1250fc831bc4ea"}
Oct 31 01:15:22.035240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:22.034948    5718 scope.go:122] "RemoveContainer" containerID="e8f3ea556052da6dae1e451319fb89f3a3f522a3bf23c07c8a2c4f0e500a0bd6"
Oct 31 01:15:22.035360 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:22.035347    5718 scope.go:122] "RemoveContainer" containerID="cfd96c187417db12fd2ca7e8ef90625d3caf4645cc55fa14be1250fc831bc4ea"
Oct 31 01:15:22.035614 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:22.035590    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn_odh-ai-gateway-infra(8fb56f72-6516-4e54-ac75-b811e416e030)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" podUID="8fb56f72-6516-4e54-ac75-b811e416e030"
Oct 31 01:15:23.162267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.162191    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"]
Oct 31 01:15:23.292062 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.292036    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:23.384335 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.384304    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca\") pod \"8fb56f72-6516-4e54-ac75-b811e416e030\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") "
Oct 31 01:15:23.384504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.384363    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7\" (UniqueName: \"kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7\") pod \"8fb56f72-6516-4e54-ac75-b811e416e030\" (UID: \"8fb56f72-6516-4e54-ac75-b811e416e030\") "
Oct 31 01:15:23.384669 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.384628    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca" pod "8fb56f72-6516-4e54-ac75-b811e416e030" (UID: "8fb56f72-6516-4e54-ac75-b811e416e030"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:23.386741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.386721    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7" pod "8fb56f72-6516-4e54-ac75-b811e416e030" (UID: "8fb56f72-6516-4e54-ac75-b811e416e030"). InnerVolumeSpecName "kube-api-access-4kcw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:23.485216 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.485128    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kcw7\" (UniqueName: \"kubernetes.io/projected/8fb56f72-6516-4e54-ac75-b811e416e030-kube-api-access-4kcw7\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:23.485216 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:23.485160    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb56f72-6516-4e54-ac75-b811e416e030-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:24.049071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:24.049036    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"
Oct 31 01:15:24.049252 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:24.049067    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn" event={"ID":"8fb56f72-6516-4e54-ac75-b811e416e030","Type":"ContainerDied","Data":"4d1e80d588c3fddb64ef88cdd310b181a82050a1444add4a8dcb05df51807d49"}
Oct 31 01:15:24.049252 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:24.049128    5718 scope.go:122] "RemoveContainer" containerID="cfd96c187417db12fd2ca7e8ef90625d3caf4645cc55fa14be1250fc831bc4ea"
Oct 31 01:15:24.215285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:24.215256    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"]
Oct 31 01:15:24.266194 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:24.266160    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f94871a1-29890155-zbzpn"]
Oct 31 01:15:25.966744 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:15:25.966714    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" path="/var/lib/kubelet/pods/8fb56f72-6516-4e54-ac75-b811e416e030/volumes"
Oct 31 01:15:37.923335 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:15:37.923292    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:16:05.389529 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.389490    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"]
Oct 31 01:16:05.390156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.389795    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" podUID="6165b123-6b3c-4fdb-9729-ad19c78cc62d" containerName="maas-api" containerID="cri-o://3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95" gracePeriod=30
Oct 31 01:16:05.649479 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.649418    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:16:05.800114 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.800064    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs\") pod \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") "
Oct 31 01:16:05.800314 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.800165    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj\" (UniqueName: \"kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj\") pod \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") "
Oct 31 01:16:05.800314 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.800248    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\") pod \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") "
Oct 31 01:16:05.800441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.800374    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca\") pod \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\" (UID: \"6165b123-6b3c-4fdb-9729-ad19c78cc62d\") "
Oct 31 01:16:05.801100 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.801068    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca" pod "6165b123-6b3c-4fdb-9729-ad19c78cc62d" (UID: "6165b123-6b3c-4fdb-9729-ad19c78cc62d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:05.802868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.802836    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj" pod "6165b123-6b3c-4fdb-9729-ad19c78cc62d" (UID: "6165b123-6b3c-4fdb-9729-ad19c78cc62d"). InnerVolumeSpecName "kube-api-access-fm4xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:05.803125 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.803064    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs" pod "6165b123-6b3c-4fdb-9729-ad19c78cc62d" (UID: "6165b123-6b3c-4fdb-9729-ad19c78cc62d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:05.803125 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.803096    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls" pod "6165b123-6b3c-4fdb-9729-ad19c78cc62d" (UID: "6165b123-6b3c-4fdb-9729-ad19c78cc62d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:05.902110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.902016    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm4xj\" (UniqueName: \"kubernetes.io/projected/6165b123-6b3c-4fdb-9729-ad19c78cc62d-kube-api-access-fm4xj\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:05.902110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.902045    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:05.902110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.902055    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6165b123-6b3c-4fdb-9729-ad19c78cc62d-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:05.902110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:05.902064    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6165b123-6b3c-4fdb-9729-ad19c78cc62d-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:06.257918 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.257884    5718 generic.go:356] "Generic (PLEG): container finished" podID="6165b123-6b3c-4fdb-9729-ad19c78cc62d" containerID="3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95" exitCode=0
Oct 31 01:16:06.258117 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.257964    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"
Oct 31 01:16:06.258117 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.257981    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" event={"ID":"6165b123-6b3c-4fdb-9729-ad19c78cc62d","Type":"ContainerDied","Data":"3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95"}
Oct 31 01:16:06.258117 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.258055    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc" event={"ID":"6165b123-6b3c-4fdb-9729-ad19c78cc62d","Type":"ContainerDied","Data":"52f9cb0d44f388864253fe32d8d61e9d640128e6d167c8eea68c91b3c30bb979"}
Oct 31 01:16:06.258117 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.258086    5718 scope.go:122] "RemoveContainer" containerID="3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95"
Oct 31 01:16:06.267431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.267410    5718 scope.go:122] "RemoveContainer" containerID="3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95"
Oct 31 01:16:06.267720 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:16:06.267699    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95\": container with ID starting with 3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95 not found: ID does not exist" containerID="3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95"
Oct 31 01:16:06.267799 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.267731    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95"} err="failed to get container status \"3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95\": rpc error: code = NotFound desc = could not find container \"3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95\": container with ID starting with 3a4653300d587f6084b4cef9faaac3acfdc380ce3bea2156d2c727c108129d95 not found: ID does not exist"
Oct 31 01:16:06.423764 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.423726    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"]
Oct 31 01:16:06.474483 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:06.474449    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b9656f65-79c688f996-nz5bc"]
Oct 31 01:16:07.925837 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:16:07.925799    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:16:07.965603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:07.965570    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6165b123-6b3c-4fdb-9729-ad19c78cc62d" path="/var/lib/kubelet/pods/6165b123-6b3c-4fdb-9729-ad19c78cc62d/volumes"
Oct 31 01:16:37.923643 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:16:37.923602    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:16:57.703029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.702951    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"]
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703351    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703361    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703370    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703375    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703388    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6165b123-6b3c-4fdb-9729-ad19c78cc62d" containerName="maas-api"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703393    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6165b123-6b3c-4fdb-9729-ad19c78cc62d" containerName="maas-api"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703408    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703413    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703469    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6165b123-6b3c-4fdb-9729-ad19c78cc62d" containerName="maas-api"
Oct 31 01:16:57.703481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703478    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703795 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703488    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.703795 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.703494    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb56f72-6516-4e54-ac75-b811e416e030" containerName="cleanup"
Oct 31 01:16:57.706732 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.706711    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.813802 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.813769    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8f6b5b-serving-cert\""
Oct 31 01:16:57.815348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.815324    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"]
Oct 31 01:16:57.817233 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.817212    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq487\" (UniqueName: \"kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.817301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.817248    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.817459 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.817437    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.817528 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.817504    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.918278 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.918230    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.918278 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.918282    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.918534 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.918405    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq487\" (UniqueName: \"kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.918534 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.918443    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.918638 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:16:57.918594    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-serving-cert: secret "maas-api-e2e-worker-w4-8f6b5b-serving-cert" not found
Oct 31 01:16:57.918713 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:16:57.918705    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls podName:b40c8b6a-e6f9-4248-9c10-ec57d60464b0 nodeName:}" failed. No retries permitted until 2026-10-31 01:16:58.418645379 +0000 UTC m=+734.581788788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls") pod "maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" (UID: "b40c8b6a-e6f9-4248-9c10-ec57d60464b0") : secret "maas-api-e2e-worker-w4-8f6b5b-serving-cert" not found
Oct 31 01:16:57.919048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.919029    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.920891 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.920865    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:57.980086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:57.980013    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq487\" (UniqueName: \"kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:58.423854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:58.423814    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:58.426518 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:58.426483    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:58.620742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:58.620705    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:16:58.998544 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:58.998522    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"]
Oct 31 01:16:59.001077 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:16:59.001046    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40c8b6a_e6f9_4248_9c10_ec57d60464b0.slice/crio-6fde1485eb0c5cf7686bd3a6937fa51332f1867d3e0c368a4e73418b12968d12 WatchSource:0}: Error finding container 6fde1485eb0c5cf7686bd3a6937fa51332f1867d3e0c368a4e73418b12968d12: Status 404 returned error can't find the container with id 6fde1485eb0c5cf7686bd3a6937fa51332f1867d3e0c368a4e73418b12968d12
Oct 31 01:16:59.544174 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:16:59.544127    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" event={"ID":"b40c8b6a-e6f9-4248-9c10-ec57d60464b0","Type":"ContainerStarted","Data":"6fde1485eb0c5cf7686bd3a6937fa51332f1867d3e0c368a4e73418b12968d12"}
Oct 31 01:17:00.551739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:00.551693    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" event={"ID":"b40c8b6a-e6f9-4248-9c10-ec57d60464b0","Type":"ContainerStarted","Data":"909b0f9c3e9cf88382218f64c213936fc400b5c010cf15515e2e852b48286d05"}
Oct 31 01:17:00.552168 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:00.551772    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:17:00.670084 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:00.670026    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" podStartSLOduration=2.800165356 podStartE2EDuration="3.670010583s" totalImagesPullingTime="869.845227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:00.668785449 +0000 UTC m=+736.831928866" watchObservedRunningTime="2026-10-31 01:17:00.670010583 +0000 UTC m=+736.833153997"
Oct 31 01:17:06.562779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:06.562750    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 01:17:07.088387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.088347    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"]
Oct 31 01:17:07.092140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.092118    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.108225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.108189    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.108403 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.108330    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.108403 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.108370    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.108502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.108413    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.108534 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.108513    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7m4v\" (UniqueName: \"kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.108595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.108577    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.148017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.147981    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"openshift-service-ca.crt\""
Oct 31 01:17:07.150042 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.150014    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"]
Oct 31 01:17:07.198087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.198047    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"default-dockercfg-k6jvm\""
Oct 31 01:17:07.198087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.198073    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"simulator-w1-baacee-kserve-self-signed-certs\""
Oct 31 01:17:07.198087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.198055    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"kube-root-ca.crt\""
Oct 31 01:17:07.209768 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.209738    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7m4v\" (UniqueName: \"kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.209909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.209784    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.209909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.209823    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.209909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.209864    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.209909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.209879    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.209909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.209902    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.210296 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.210274    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.210353 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.210293    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.210558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.210536    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.212850 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.212827    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.213029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.213001    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.269109 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.269073    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7m4v\" (UniqueName: \"kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v\") pod \"simulator-w1-baacee-kserve-76cf886d9c-6n59n\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") " pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.406343 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.406249    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:07.779963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:07.779941    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"]
Oct 31 01:17:07.782231 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:17:07.782197    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36cded3f_8e56_4db3_b3e9_48f9b1bd7df5.slice/crio-6a045853d5e42f605b1eb92c4a98c4a5903e7694883e33cffabccebe69f84eb6 WatchSource:0}: Error finding container 6a045853d5e42f605b1eb92c4a98c4a5903e7694883e33cffabccebe69f84eb6: Status 404 returned error can't find the container with id 6a045853d5e42f605b1eb92c4a98c4a5903e7694883e33cffabccebe69f84eb6
Oct 31 01:17:07.926293 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:17:07.926253    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:17:08.591067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:08.591031    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" event={"ID":"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5","Type":"ContainerStarted","Data":"6a045853d5e42f605b1eb92c4a98c4a5903e7694883e33cffabccebe69f84eb6"}
Oct 31 01:17:10.602504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:10.602467    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" event={"ID":"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5","Type":"ContainerStarted","Data":"d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2"}
Oct 31 01:17:10.723145 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:10.723086    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" podStartSLOduration=1.5415827 podStartE2EDuration="3.723070481s" totalImagesPullingTime="2.181487781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:10.722331552 +0000 UTC m=+746.885474966" watchObservedRunningTime="2026-10-31 01:17:10.723070481 +0000 UTC m=+746.886213895"
Oct 31 01:17:11.407059 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:11.407030    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:17.406787 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:17.406736    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:17.419953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:17.419924    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:17.657264 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:17.657182    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 01:17:32.388004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.387967    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"]
Oct 31 01:17:32.391926 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.391906    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.448320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.448287    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"]
Oct 31 01:17:32.497240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.497207    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"premium-w3-a8e157-kserve-self-signed-certs\""
Oct 31 01:17:32.497240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.497226    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"default-dockercfg-rlpbk\""
Oct 31 01:17:32.497240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.497219    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"openshift-service-ca.crt\""
Oct 31 01:17:32.498449 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.498424    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"kube-root-ca.crt\""
Oct 31 01:17:32.559116 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.559086    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.559306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.559125    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.559306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.559155    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.559388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.559298    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.559388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.559341    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.559469 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.559396    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kq7j\" (UniqueName: \"kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.660786 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.660689    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.660786 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.660744    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661015 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.660795    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661015 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.660837    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661015 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.660881    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661015 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.660913    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kq7j\" (UniqueName: \"kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.661204    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661283 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.661256    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.661432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.661412    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.663399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.663369    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.663875 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.663859    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:32.719390 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:32.719355    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kq7j\" (UniqueName: \"kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j\") pod \"premium-w3-a8e157-kserve-7f9778d54-xx79p\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") " pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:33.003619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:33.003582    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:33.376275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:33.376246    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"]
Oct 31 01:17:33.378379 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:17:33.378353    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod815614a9_1038_4ad5_aed5_057bece8b655.slice/crio-a9c79691fdcb10e614329c02724d8cd68f118c4429e684251fc5785552efcee9 WatchSource:0}: Error finding container a9c79691fdcb10e614329c02724d8cd68f118c4429e684251fc5785552efcee9: Status 404 returned error can't find the container with id a9c79691fdcb10e614329c02724d8cd68f118c4429e684251fc5785552efcee9
Oct 31 01:17:33.728763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:33.727811    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" event={"ID":"815614a9-1038-4ad5-aed5-057bece8b655","Type":"ContainerStarted","Data":"6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59"}
Oct 31 01:17:33.728763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:33.727874    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" event={"ID":"815614a9-1038-4ad5-aed5-057bece8b655","Type":"ContainerStarted","Data":"a9c79691fdcb10e614329c02724d8cd68f118c4429e684251fc5785552efcee9"}
Oct 31 01:17:33.847862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:33.847795    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" podStartSLOduration=1.847781106 podStartE2EDuration="1.847781106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:33.846077939 +0000 UTC m=+770.009221355" watchObservedRunningTime="2026-10-31 01:17:33.847781106 +0000 UTC m=+770.010924520"
Oct 31 01:17:34.004347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:34.004316    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:37.924694 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:17:37.924630    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:17:43.004423 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:43.004376    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:43.018220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:43.018193    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:17:43.788308 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:17:43.788279    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 01:18:07.924288 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:18:07.924246    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:18:18.683240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.683197    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"]
Oct 31 01:18:18.687235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.687211    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.713965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.713920    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.714119 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.714103    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.714185 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.714140    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.714286 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.714259    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.714397 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.714381    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.714456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.714428    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfq4\" (UniqueName: \"kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.742785 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.742754    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"openshift-service-ca.crt\""
Oct 31 01:18:18.793212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.793140    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"kube-root-ca.crt\""
Oct 31 01:18:18.793212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.793182    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"default-dockercfg-s766l\""
Oct 31 01:18:18.793458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.793418    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"simulator-w0-057e2d-kserve-self-signed-certs\""
Oct 31 01:18:18.795305 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.795283    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"]
Oct 31 01:18:18.815665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.815630    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfq4\" (UniqueName: \"kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.815800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.815694    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.815800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.815750    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.815800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.815774    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.815974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.815820    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.815974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.815891    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.816183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.816162    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.816242 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.816198    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.816296 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.816267    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.818388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.818360    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.818545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.818528    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:18.874276 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:18.874241    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfq4\" (UniqueName: \"kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4\") pod \"simulator-w0-057e2d-kserve-65d79c7564-pj96l\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") " pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:19.028062 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:19.028021    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:19.405390 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:19.405333    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"]
Oct 31 01:18:19.407747 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:18:19.407719    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5455c3c_e54d_4088_a012_688845de714a.slice/crio-7cb2df619a52b8a95fb84482c2a80a9178df9d2f315b9fba3646a4b4dffd78c9 WatchSource:0}: Error finding container 7cb2df619a52b8a95fb84482c2a80a9178df9d2f315b9fba3646a4b4dffd78c9: Status 404 returned error can't find the container with id 7cb2df619a52b8a95fb84482c2a80a9178df9d2f315b9fba3646a4b4dffd78c9
Oct 31 01:18:19.971800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:19.971752    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" event={"ID":"b5455c3c-e54d-4088-a012-688845de714a","Type":"ContainerStarted","Data":"138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c"}
Oct 31 01:18:19.972289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:19.971814    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" event={"ID":"b5455c3c-e54d-4088-a012-688845de714a","Type":"ContainerStarted","Data":"7cb2df619a52b8a95fb84482c2a80a9178df9d2f315b9fba3646a4b4dffd78c9"}
Oct 31 01:18:20.028331 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:20.028302    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:20.091183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:20.091125    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" podStartSLOduration=2.091110685 podStartE2EDuration="2.091110685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:20.089861932 +0000 UTC m=+816.253005347" watchObservedRunningTime="2026-10-31 01:18:20.091110685 +0000 UTC m=+816.254254100"
Oct 31 01:18:29.028480 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:29.028445    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:29.041632 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:29.041600    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:30.029801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:18:30.029773    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:18:37.923353 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:18:37.923313    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:19:07.924264 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:19:07.924221    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:19:32.664637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.664604    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"]
Oct 31 01:19:32.672283 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.670776    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.765197 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.765164    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.765399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.765231    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.765399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.765325    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6wr\" (UniqueName: \"kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.765399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.765375    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.765591 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.765437    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.765591 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.765489    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.775487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.775459    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"premium-w0-057e2d-kserve-self-signed-certs\""
Oct 31 01:19:32.777146 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.777125    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"]
Oct 31 01:19:32.866863 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.866832    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.866876    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6wr\" (UniqueName: \"kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.866915    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.866974    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.867017    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.867064    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867424 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.867400    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867523 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.867504    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.867575 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.867526    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.869336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.869310    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.869574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.869554    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.925984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.925903    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6wr\" (UniqueName: \"kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr\") pod \"premium-w0-057e2d-kserve-6bcfbd54d9-6d92f\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") " pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:32.984870 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:32.984829    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:33.384479 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:33.384448    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"]
Oct 31 01:19:33.386166 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:19:33.386138    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54f8ef22_0f82_4bb7_aea2_23269ded877d.slice/crio-3bd7651d24edd55423decf8f1daddff47356ae5a119146edfadcc5613b7a2442 WatchSource:0}: Error finding container 3bd7651d24edd55423decf8f1daddff47356ae5a119146edfadcc5613b7a2442: Status 404 returned error can't find the container with id 3bd7651d24edd55423decf8f1daddff47356ae5a119146edfadcc5613b7a2442
Oct 31 01:19:34.347976 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:34.347936    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" event={"ID":"54f8ef22-0f82-4bb7-aea2-23269ded877d","Type":"ContainerStarted","Data":"a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e"}
Oct 31 01:19:34.347976 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:34.347980    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" event={"ID":"54f8ef22-0f82-4bb7-aea2-23269ded877d","Type":"ContainerStarted","Data":"3bd7651d24edd55423decf8f1daddff47356ae5a119146edfadcc5613b7a2442"}
Oct 31 01:19:34.469273 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:34.469209    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" podStartSLOduration=2.469194625 podStartE2EDuration="2.469194625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:34.468115748 +0000 UTC m=+890.631259166" watchObservedRunningTime="2026-10-31 01:19:34.469194625 +0000 UTC m=+890.632338040"
Oct 31 01:19:34.985483 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:34.985441    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:37.923104 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:19:37.923058    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:19:42.985048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:42.985009    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:42.998151 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:42.998126    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:19:43.406963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:19:43.406935    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:20:07.926172 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:20:07.926094    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:20:37.924561 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:20:37.924522    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:20:44.041567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.041524    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"]
Oct 31 01:20:44.045772 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.045749    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198476 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198441    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198488    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198720 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198634    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198756 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198718    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcpd\" (UniqueName: \"kubernetes.io/projected/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-kube-api-access-xzcpd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198756 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198748    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198767    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198857 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198815    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.198927 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198911    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.199005 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.198990    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.246035 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.246003    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"]
Oct 31 01:20:44.300559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300472    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300528    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300559    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300583    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcpd\" (UniqueName: \"kubernetes.io/projected/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-kube-api-access-xzcpd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300609    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300632    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300721    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300752    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.300890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.300810    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.301206 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.301179    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.301268 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.301225    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.301322 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.301284    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.301388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.301366    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.301432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.301422    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.303113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.303090    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.303321 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.303301    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.358005 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.357973    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.358187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.358044    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcpd\" (UniqueName: \"kubernetes.io/projected/39645c3b-b2f4-448a-8683-c7b5b4ec1e51-kube-api-access-xzcpd\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-9mzcr\" (UID: \"39645c3b-b2f4-448a-8683-c7b5b4ec1e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.358806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.358791    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:44.786564 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.786539    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"]
Oct 31 01:20:44.788480 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:20:44.788448    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39645c3b_b2f4_448a_8683_c7b5b4ec1e51.slice/crio-5adc0e84b38e74693faf23623642d312cbc20f2abc3170f278e196964724297f WatchSource:0}: Error finding container 5adc0e84b38e74693faf23623642d312cbc20f2abc3170f278e196964724297f: Status 404 returned error can't find the container with id 5adc0e84b38e74693faf23623642d312cbc20f2abc3170f278e196964724297f
Oct 31 01:20:44.790746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.790708    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:44.790849 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.790786    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:44.790849 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:44.790815    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:45.767582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:45.767538    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" event={"ID":"39645c3b-b2f4-448a-8683-c7b5b4ec1e51","Type":"ContainerStarted","Data":"49dba970bc0b1fde5d5107e7782467bde6741de8706fe841faddc1570edb7f90"}
Oct 31 01:20:45.768063 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:45.767615    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" event={"ID":"39645c3b-b2f4-448a-8683-c7b5b4ec1e51","Type":"ContainerStarted","Data":"5adc0e84b38e74693faf23623642d312cbc20f2abc3170f278e196964724297f"}
Oct 31 01:20:45.991094 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:45.991034    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podStartSLOduration=2.99101411 podStartE2EDuration="2.99101411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:45.939519725 +0000 UTC m=+962.102663142" watchObservedRunningTime="2026-10-31 01:20:45.99101411 +0000 UTC m=+962.154157524"
Oct 31 01:20:46.363854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:46.362069    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:46.368153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:46.368120    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 01:20:46.368337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:46.368183    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 01:20:47.362119 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:47.362076    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.363075 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:47.363039    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 01:20:48.366434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:48.365868    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.366434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:48.365933    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 01:20:48.904302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:48.904265    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"]
Oct 31 01:20:48.908150 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:48.908134    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:48.973719 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:48.972620    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"]
Oct 31 01:20:49.014537 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.014509    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"distinct-w0-057e2d-kserve-self-signed-certs\""
Oct 31 01:20:49.070298 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.070264    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.070516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.070319    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.070516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.070357    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.070516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.070380    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.070516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.070433    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9pd\" (UniqueName: \"kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.070516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.070508    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.171740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.171228    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.171740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.171333    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.171740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.171373    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.171740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.171413    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.171740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.171443    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.171740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.171504    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9pd\" (UniqueName: \"kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.172602 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.172568    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.172984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.172910    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.172984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.172925    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.174983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.174957    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.175882 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.175861    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.229612 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.229580    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9pd\" (UniqueName: \"kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd\") pod \"distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.363151 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.363111    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.363416 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.363386    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 01:20:49.520855 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.520820    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:49.923104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:49.923072    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"]
Oct 31 01:20:49.925178 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:20:49.925152    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3277d75c_b24f_4a79_987e_a2128749a7c7.slice/crio-bf41cfc439eb9a776a2660c19c1a607b787e6224035ff70b9c094cccfb29a54b WatchSource:0}: Error finding container bf41cfc439eb9a776a2660c19c1a607b787e6224035ff70b9c094cccfb29a54b: Status 404 returned error can't find the container with id bf41cfc439eb9a776a2660c19c1a607b787e6224035ff70b9c094cccfb29a54b
Oct 31 01:20:50.365680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:50.365332    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.365680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:50.365391    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 01:20:50.801469 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:50.801430    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" event={"ID":"3277d75c-b24f-4a79-987e-a2128749a7c7","Type":"ContainerStarted","Data":"337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905"}
Oct 31 01:20:50.801469 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:50.801472    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" event={"ID":"3277d75c-b24f-4a79-987e-a2128749a7c7","Type":"ContainerStarted","Data":"bf41cfc439eb9a776a2660c19c1a607b787e6224035ff70b9c094cccfb29a54b"}
Oct 31 01:20:51.360126 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:51.360091    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.360342 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:51.360151    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 01:20:51.521384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:51.521351    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:53.171618 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.171574    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:53.172135 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.172048    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:53.172987 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.172964    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr"
Oct 31 01:20:53.341978 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.341923    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" podStartSLOduration=5.341909789 podStartE2EDuration="5.341909789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:50.918775393 +0000 UTC m=+967.081918808" watchObservedRunningTime="2026-10-31 01:20:53.341909789 +0000 UTC m=+969.505053232"
Oct 31 01:20:53.566581 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.566544    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"]
Oct 31 01:20:53.570558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.570536    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.615957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.615929    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616126 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.615963    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616126 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.615989    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.616141    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.616164    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.616201    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616323 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.616286    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616323 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.616318    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.616424 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.616400    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6sq\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.623358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.623338    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a8e157-openshift-default-dockercfg-f24hn\""
Oct 31 01:20:53.629380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.629353    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"]
Oct 31 01:20:53.717344 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717304    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717372    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717392    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717421    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717437    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717463    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717507    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6sq\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717594    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717619    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717829    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.717925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717878    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.718145 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.717938    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.718145 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.718066    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.718236 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.718149    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.720166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.720134    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.720359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.720343    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.778021 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.777988    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6sq\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.780102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.780077    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:53.887147 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:53.887061    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:54.313330 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.313304    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"]
Oct 31 01:20:54.314691 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:20:54.314665    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cdfe7f_f41c_4a1d_b947_75e324c0ea23.slice/crio-b3718517d03b77fe9a9e38d35d79e08e8e2e6fb9c023e875fe65c2eb49bf1727 WatchSource:0}: Error finding container b3718517d03b77fe9a9e38d35d79e08e8e2e6fb9c023e875fe65c2eb49bf1727: Status 404 returned error can't find the container with id b3718517d03b77fe9a9e38d35d79e08e8e2e6fb9c023e875fe65c2eb49bf1727
Oct 31 01:20:54.317293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.317254    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:54.317431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.317340    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:54.317431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.317381    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:54.824262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.824224    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" event={"ID":"93cdfe7f-f41c-4a1d-b947-75e324c0ea23","Type":"ContainerStarted","Data":"700009f055edc0303ae8b07bcacccfbd9aa414cc48f99ae0f61c0864d0b9baac"}
Oct 31 01:20:54.824450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.824333    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" event={"ID":"93cdfe7f-f41c-4a1d-b947-75e324c0ea23","Type":"ContainerStarted","Data":"b3718517d03b77fe9a9e38d35d79e08e8e2e6fb9c023e875fe65c2eb49bf1727"}
Oct 31 01:20:54.887235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.887200    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:20:54.995511 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:54.995439    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podStartSLOduration=1.995416522 podStartE2EDuration="1.995416522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:54.994151979 +0000 UTC m=+971.157295434" watchObservedRunningTime="2026-10-31 01:20:54.995416522 +0000 UTC m=+971.158559937"
Oct 31 01:20:55.888913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:55.888874    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:20:55.889400 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:55.888947    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:20:56.887831 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:56.887794    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:20:56.888020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:56.887845    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:20:57.888190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:57.888154    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:20:57.888598 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:57.888209    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:20:58.887507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:58.887470    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:20:58.887821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:58.887535    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:20:59.165017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.164922    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"]
Oct 31 01:20:59.170179 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.170149    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197366 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197335    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197510 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197387    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197510 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197407    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197581 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197551    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197597    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197631    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197759 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197745    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5btr\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197797 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197774    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.197875 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.197846    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.261674 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.261258    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"]
Oct 31 01:20:59.299204 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299170    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299221    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299249    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299286    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299333    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299639    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299751 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299680    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299810    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299888 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299873    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5btr\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299937 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299915    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.299937 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299924    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.300011 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.299993    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.300297 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.300276    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.301002 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.300976    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.301942 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.301913    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.303043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.302727    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.367865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.367821    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5btr\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.368271 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.368247    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-7f274\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.483640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.483559    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:20:59.521733 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.521693    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:59.534955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.534929    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:59.886224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.886191    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:20:59.887578 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.887554    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:20:59.887718 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.887600    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:20:59.929700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.929669    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"]
Oct 31 01:20:59.931319 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:20:59.931289    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5314ae_9bd7_44c1_9345_73dddd0b6723.slice/crio-0ba0f06da4a28b8f21fffda65c94b9822fc480cc54743ca4e590f41a09d51af7 WatchSource:0}: Error finding container 0ba0f06da4a28b8f21fffda65c94b9822fc480cc54743ca4e590f41a09d51af7: Status 404 returned error can't find the container with id 0ba0f06da4a28b8f21fffda65c94b9822fc480cc54743ca4e590f41a09d51af7
Oct 31 01:20:59.933641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.933612    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:59.933730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.933705    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:20:59.933766 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:20:59.933736    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:21:00.882503 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:00.882467    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" event={"ID":"2d5314ae-9bd7-44c1-9345-73dddd0b6723","Type":"ContainerStarted","Data":"4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950"}
Oct 31 01:21:00.883011 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:00.882522    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" event={"ID":"2d5314ae-9bd7-44c1-9345-73dddd0b6723","Type":"ContainerStarted","Data":"0ba0f06da4a28b8f21fffda65c94b9822fc480cc54743ca4e590f41a09d51af7"}
Oct 31 01:21:00.887759 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:00.887729    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 01:21:00.887886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:00.887797    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 01:21:01.098868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.098810    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podStartSLOduration=2.098794346 podStartE2EDuration="2.098794346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:01.062137274 +0000 UTC m=+977.225280690" watchObservedRunningTime="2026-10-31 01:21:01.098794346 +0000 UTC m=+977.261937761"
Oct 31 01:21:01.483858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.483814    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:21:01.485294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.485272    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:01.485361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.485333    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:01.891800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.891774    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:21:01.892268 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.892200    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:21:01.893007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:01.892987    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:21:02.484213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:02.484176    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:02.484415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:02.484263    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:03.484746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:03.484701    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:03.485175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:03.484771    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:04.484852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:04.484812    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:04.485364 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:04.484882    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:05.484422 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:05.484384    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:05.484680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:05.484442    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:06.485025 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:06.484986    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:06.485447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:06.485047    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:07.484122 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:07.484083    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:21:07.484311 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:07.484176    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 31 01:21:07.925776 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:21:07.925739    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:21:08.539518 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:08.539487    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:21:08.539900 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:08.539876    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:21:08.541157 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:08.541135    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:21:37.923991 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:21:37.923948    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:21:50.262162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.262124    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"]
Oct 31 01:21:50.267455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.267432    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.319016 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.318975    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.319200 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.319051    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.319200 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.319133    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.319288 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.319264    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.319326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.319305    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.319361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.319340    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxqp\" (UniqueName: \"kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.371007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.370966    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057e2d\"/\"distinct-2-w0-057e2d-kserve-self-signed-certs\""
Oct 31 01:21:50.373179 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.373156    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"]
Oct 31 01:21:50.420607 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.420566    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.420607 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.420604    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.420894 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.420692    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.420894 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.420796    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.420894 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.420822    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.420894 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.420857    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxqp\" (UniqueName: \"kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.421117 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.421050    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.421185 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.421134    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.421185 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.421146    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.423366 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.423334    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.423606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.423586    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.479131 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.479093    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxqp\" (UniqueName: \"kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp\") pod \"distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") " pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.579992 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.579897    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:21:50.974183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:50.974153    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"]
Oct 31 01:21:50.975926 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:21:50.975898    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa64d50_608d_4a3d_b5f8_c630ef9c23ff.slice/crio-85590edd1ad5d7775fc2a6552aa68415fc4cbb548b810e242ad8686b5dcf128c WatchSource:0}: Error finding container 85590edd1ad5d7775fc2a6552aa68415fc4cbb548b810e242ad8686b5dcf128c: Status 404 returned error can't find the container with id 85590edd1ad5d7775fc2a6552aa68415fc4cbb548b810e242ad8686b5dcf128c
Oct 31 01:21:51.177086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:51.177052    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" event={"ID":"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff","Type":"ContainerStarted","Data":"0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec"}
Oct 31 01:21:51.177274 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:51.177097    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" event={"ID":"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff","Type":"ContainerStarted","Data":"85590edd1ad5d7775fc2a6552aa68415fc4cbb548b810e242ad8686b5dcf128c"}
Oct 31 01:21:51.301458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:51.301391    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" podStartSLOduration=1.301371435 podStartE2EDuration="1.301371435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:51.300989062 +0000 UTC m=+1027.464132522" watchObservedRunningTime="2026-10-31 01:21:51.301371435 +0000 UTC m=+1027.464514855"
Oct 31 01:21:51.580232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:21:51.580146    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:22:00.580140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:00.580104    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:22:00.593388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:00.593362    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:22:01.240729 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:01.240694    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:22:07.925001 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:22:07.924892    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:22:09.191186 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.191152    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"]
Oct 31 01:22:09.194868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.194849    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.326836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.326793    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327016 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.326912    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327016 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327005    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327124 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327024    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327124 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327062    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327124 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327082    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327254 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327235    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327285    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxmf\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.327356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.327344    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.405007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.404975    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"]
Oct 31 01:22:09.428575 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428536    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.428790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428595    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxmf\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.428790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428626    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.428790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428679    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.428790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428756    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428823    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428848    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428892    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.428914    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.429017    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429457 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.429434    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.429749    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.429900 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.429809    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.430133 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.430061    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.431584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.431556    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.431781 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.431761    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.488949 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.488879    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.489071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.489022    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxmf\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.510520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.510496    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:09.939067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.939034    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"]
Oct 31 01:22:09.942731 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:22:09.942701    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf096d4aa_8b79_4d52_b5fd_bdb7ac6b2dd2.slice/crio-3b7efb5504c1fbe00b67766abaaea1e4ccff6bfd5e3ca9c0e657d324c35da577 WatchSource:0}: Error finding container 3b7efb5504c1fbe00b67766abaaea1e4ccff6bfd5e3ca9c0e657d324c35da577: Status 404 returned error can't find the container with id 3b7efb5504c1fbe00b67766abaaea1e4ccff6bfd5e3ca9c0e657d324c35da577
Oct 31 01:22:09.944889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.944851    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:22:09.945003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.944936    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:22:09.945003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:09.944979    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:22:10.120362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.120320    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"]
Oct 31 01:22:10.124166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.124148    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.177004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.176974    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8f6b5b-openshift-default-dockercfg-6t9gt\""
Oct 31 01:22:10.183909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.183878    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"]
Oct 31 01:22:10.238835 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.238735    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.238835 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.238782    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.238835 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.238813    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kt2n\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.239312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.238967    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.239312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.239024    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.239312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.239164    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.239312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.239220    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.239464 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.239318    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.239464 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.239426    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.283712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.283673    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" event={"ID":"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2","Type":"ContainerStarted","Data":"be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8"}
Oct 31 01:22:10.283914 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.283739    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" event={"ID":"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2","Type":"ContainerStarted","Data":"3b7efb5504c1fbe00b67766abaaea1e4ccff6bfd5e3ca9c0e657d324c35da577"}
Oct 31 01:22:10.340491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340454    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340518    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340703    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340789    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340837    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340855    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340885    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kt2n\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.340928 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340922    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.341135 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.340965    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.341194 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.341128    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.341194 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.341157    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.341462 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.341245    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.341462 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.341453    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.342057 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.342031    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.343120 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.343093    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.343306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.343290    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.399405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.399365    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.399563 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.399496    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kt2n\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.406349 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.406298    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podStartSLOduration=1.406284791 podStartE2EDuration="1.406284791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:10.404329486 +0000 UTC m=+1046.567472901" watchObservedRunningTime="2026-10-31 01:22:10.406284791 +0000 UTC m=+1046.569428206"
Oct 31 01:22:10.439954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.439911    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:10.511437 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.511365    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:10.515535 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.515503    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:10.515690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.515554    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:10.889242 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.889208    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"]
Oct 31 01:22:10.890764 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:22:10.890732    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c783693_7a55_42cf_a4c6_983a680377e9.slice/crio-72ae01fd3289abfcab62c6013fec2963ebd4718873011010c6769f365e5ac1c4 WatchSource:0}: Error finding container 72ae01fd3289abfcab62c6013fec2963ebd4718873011010c6769f365e5ac1c4: Status 404 returned error can't find the container with id 72ae01fd3289abfcab62c6013fec2963ebd4718873011010c6769f365e5ac1c4
Oct 31 01:22:10.893142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.893113    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:22:10.893233 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.893173    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:22:10.893233 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:10.893204    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:22:11.292743 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.292695    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" event={"ID":"6c783693-7a55-42cf-a4c6-983a680377e9","Type":"ContainerStarted","Data":"2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc"}
Oct 31 01:22:11.293167 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.292770    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" event={"ID":"6c783693-7a55-42cf-a4c6-983a680377e9","Type":"ContainerStarted","Data":"72ae01fd3289abfcab62c6013fec2963ebd4718873011010c6769f365e5ac1c4"}
Oct 31 01:22:11.427520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.427463    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podStartSLOduration=1.4274456660000001 podStartE2EDuration="1.427445666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:11.427297215 +0000 UTC m=+1047.590440630" watchObservedRunningTime="2026-10-31 01:22:11.427445666 +0000 UTC m=+1047.590589081"
Oct 31 01:22:11.454681 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.452016    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:11.465843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.465803    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:11.466060 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.465876    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:11.511796 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.511761    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:11.511965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:11.511815    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:12.452682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:12.449126    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:12.452682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:12.449206    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:12.511094 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:12.511062    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:12.511266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:12.511126    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:13.448680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:13.445046    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:13.448680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:13.445123    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:13.511342 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:13.511303    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:13.511717 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:13.511351    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:14.461676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:14.449055    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:14.461676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:14.449117    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:14.511063 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:14.511020    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:14.511258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:14.511076    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:15.450594 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:15.450138    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:15.450594 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:15.450198    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:15.511213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:15.511184    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:15.511458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:15.511433    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:16.445907 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:16.445864    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:16.446601 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:16.446568    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:16.511468 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:16.511432    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 31 01:22:16.511886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:16.511521    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 31 01:22:17.456679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:17.452008    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:22:17.456679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:17.452068    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 31 01:22:17.515807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:17.515774    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:17.516269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:17.516036    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:17.516766 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:17.516745    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:22:18.554045 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:18.554005    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:18.554559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:18.554459    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:18.555308 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:22:18.555277    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 01:22:37.951086 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:22:37.951044    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:23:05.786641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.786555    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"]
Oct 31 01:23:05.791043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.791009    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878125 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878095    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878330 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878136    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878330 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878307    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878425 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878412    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878468 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878456    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878510 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878479    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878550 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878522    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878582    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.878698 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.878641    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl9xq\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.894889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.894859    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057e2d-openshift-default-dockercfg-2bvxk\""
Oct 31 01:23:05.897454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.897427    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"]
Oct 31 01:23:05.979597 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979563    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979788 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979636    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979788 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979727    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979788 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979754    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979788 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979775    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979822    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979861    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl9xq\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979911    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.979983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.979949    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.980188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.980045    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.980188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.980150    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.980282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.980198    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.980617 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.980587    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.980954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.980712    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.982517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.982492    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:05.982630 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:05.982612    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:06.037824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.037757    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl9xq\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:06.037824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.037760    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:06.106464 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.106419    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:06.535225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.535198    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"]
Oct 31 01:23:06.536927 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:23:06.536897    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c06b2b8_2623_4b3d_9558_ed97794b2f9c.slice/crio-1479df07e3516072a67a7a66afb1f22a880d3d194a8aef80ff747ac9c38f93d4 WatchSource:0}: Error finding container 1479df07e3516072a67a7a66afb1f22a880d3d194a8aef80ff747ac9c38f93d4: Status 404 returned error can't find the container with id 1479df07e3516072a67a7a66afb1f22a880d3d194a8aef80ff747ac9c38f93d4
Oct 31 01:23:06.539408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.539376    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:23:06.539495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.539441    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:23:06.539495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.539472    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:23:06.634671 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.634620    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" event={"ID":"2c06b2b8-2623-4b3d-9558-ed97794b2f9c","Type":"ContainerStarted","Data":"8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba"}
Oct 31 01:23:06.634841 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.634709    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" event={"ID":"2c06b2b8-2623-4b3d-9558-ed97794b2f9c","Type":"ContainerStarted","Data":"1479df07e3516072a67a7a66afb1f22a880d3d194a8aef80ff747ac9c38f93d4"}
Oct 31 01:23:06.755041 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:06.754986    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podStartSLOduration=1.754972347 podStartE2EDuration="1.754972347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:06.753944887 +0000 UTC m=+1102.917088307" watchObservedRunningTime="2026-10-31 01:23:06.754972347 +0000 UTC m=+1102.918115762"
Oct 31 01:23:07.107583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:07.107551    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:07.109310 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:07.109272    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:07.109452 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:07.109329    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:07.924334 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:23:07.924298    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:23:08.107306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:08.107274    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:08.107493 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:08.107327    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:09.107267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.107232    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:09.107819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.107295    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:09.662678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.658573    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"]
Oct 31 01:23:09.682907 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.682872    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730099    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730173    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730271    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730308    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730376    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730452    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730481    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730723 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730546    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.730723 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.730583    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.795321 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.795276    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"]
Oct 31 01:23:09.831852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.831815    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.831852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.831867    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.832137 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.831959    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.832137 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.831999    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.832137 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.832067    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.832304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.832143    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.832743 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.832707    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.833001 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.832977    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.834281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.834259    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.834574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.833417    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.834574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.833751    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.834574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.833205    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.837724 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.837696    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.838049 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.838025    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.839238 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.839210    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.898434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.898397    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.899740 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.899282    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:09.902320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:09.902285    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:10.004340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.004305    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:10.107807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.107741    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:10.108207 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.107809    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:10.515987 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.509208    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"]
Oct 31 01:23:10.546933 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.545551    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:23:10.546933 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.545709    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:23:10.546933 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.545764    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:23:10.723680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:10.722707    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" event={"ID":"a6fdc068-5f83-45e0-8655-3bfb0040e0a4","Type":"ContainerStarted","Data":"f5fd88cb5df18f21ac974f7de574d0526da80fe3e1755332c3e4c1352abdb404"}
Oct 31 01:23:11.107309 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:11.107198    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:11.107644 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:11.107607    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:11.840571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:11.840526    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" event={"ID":"a6fdc068-5f83-45e0-8655-3bfb0040e0a4","Type":"ContainerStarted","Data":"cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab"}
Oct 31 01:23:11.992093 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:11.992032    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podStartSLOduration=2.992013258 podStartE2EDuration="2.992013258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:11.990261805 +0000 UTC m=+1108.153405223" watchObservedRunningTime="2026-10-31 01:23:11.992013258 +0000 UTC m=+1108.155156674"
Oct 31 01:23:12.007681 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:12.006597    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:12.008026 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:12.007996    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:12.008231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:12.008198    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:12.108929 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:12.108825    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:12.109217 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:12.109171    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:13.011680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:13.005204    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:13.011680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:13.005287    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:13.107679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:13.107056    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:13.107679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:13.107125    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:14.005325 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:14.005289    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:14.005512 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:14.005363    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:14.107000 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:14.106960    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:23:14.107393 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:14.107028    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 31 01:23:15.005466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:15.005427    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:15.005682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:15.005491    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:16.005637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.005590    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:16.005637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.005669    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:16.109399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.109351    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:23:16.109573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.109430    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:23:16.126335 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.126299    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:16.126946 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.126912    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:16.127647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:16.127630    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:23:17.005189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:17.005157    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:17.005368 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:17.005207    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:18.005763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:18.005727    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:23:18.006163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:18.005788    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:23:19.072086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:19.072054    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:19.072556 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:19.072461    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:19.073467 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:19.073445    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:23:37.923312 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:23:37.923271    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:23:53.211085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.211042    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"]
Oct 31 01:23:53.216627 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.216602    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.256626 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.256590    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.256858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.256834    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89rpz\" (UniqueName: \"kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.256940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.256899    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.257007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.256976    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.257074 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.257059    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.257130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.257117    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.272396 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.272365    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"distinct-2-w1-baacee-kserve-self-signed-certs\""
Oct 31 01:23:53.324361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.324324    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"]
Oct 31 01:23:53.358211 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358174    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358391 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358233    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358391 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358309    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358506 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358392    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89rpz\" (UniqueName: \"kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358506 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358434    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358608 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358503    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358792    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358846    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.358974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.358916    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.360900 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.360877    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.361127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.361105    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.416462 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.416427    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89rpz\" (UniqueName: \"kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz\") pod \"distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.529163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.529120    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:23:53.919504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:53.919477    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"]
Oct 31 01:23:53.921185 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:23:53.921160    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb9b7ddb_7122_4d82_8fd7_6e7148f29a8d.slice/crio-73809208da53cf40c6a5231df176a35fb2ca5eeaac21b38082c535d9a5b81d1a WatchSource:0}: Error finding container 73809208da53cf40c6a5231df176a35fb2ca5eeaac21b38082c535d9a5b81d1a: Status 404 returned error can't find the container with id 73809208da53cf40c6a5231df176a35fb2ca5eeaac21b38082c535d9a5b81d1a
Oct 31 01:23:54.169428 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:54.169339    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" event={"ID":"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d","Type":"ContainerStarted","Data":"25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f"}
Oct 31 01:23:54.169428 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:54.169392    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" event={"ID":"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d","Type":"ContainerStarted","Data":"73809208da53cf40c6a5231df176a35fb2ca5eeaac21b38082c535d9a5b81d1a"}
Oct 31 01:23:54.286368 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:54.286314    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podStartSLOduration=1.286300744 podStartE2EDuration="1.286300744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:54.284973606 +0000 UTC m=+1150.448117019" watchObservedRunningTime="2026-10-31 01:23:54.286300744 +0000 UTC m=+1150.449444159"
Oct 31 01:23:54.530131 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:23:54.530094    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:24:03.530204 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:03.530168    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:24:03.542924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:03.542898    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:24:04.360794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:04.360765    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:24:06.270518 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.270488    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"]
Oct 31 01:24:06.274680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.274639    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.381431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.381392    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"]
Oct 31 01:24:06.405044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405000    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405055    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405263 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405216    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405310 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405276    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405417 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405393    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405518 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405503    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405575    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krpnw\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405782 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405766    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.405837 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.405798    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507053 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507015    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507053 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507061    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507097    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507114    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507168    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507197    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507226    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507305    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507343    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krpnw\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507485    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507577    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507616    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507882    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.507941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.507902    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.509838 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.509815    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.509955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.509908    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.565799 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.565727    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.565966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.565925    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krpnw\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:06.588932 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:06.588908    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:07.017914 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.017877    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"]
Oct 31 01:24:07.020015 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:24:07.019985    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5bb83a_d12b_4dc9_9365_c1050a60914b.slice/crio-32514c5d54683cfbbe97f64eb17c54c9f57c0a1a1050e2ab043241acd839876e WatchSource:0}: Error finding container 32514c5d54683cfbbe97f64eb17c54c9f57c0a1a1050e2ab043241acd839876e: Status 404 returned error can't find the container with id 32514c5d54683cfbbe97f64eb17c54c9f57c0a1a1050e2ab043241acd839876e
Oct 31 01:24:07.022608 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.022574    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:07.022714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.022669    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:07.022714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.022701    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:07.369213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.369123    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" event={"ID":"4e5bb83a-d12b-4dc9-9365-c1050a60914b","Type":"ContainerStarted","Data":"a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe"}
Oct 31 01:24:07.369213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.369182    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" event={"ID":"4e5bb83a-d12b-4dc9-9365-c1050a60914b","Type":"ContainerStarted","Data":"32514c5d54683cfbbe97f64eb17c54c9f57c0a1a1050e2ab043241acd839876e"}
Oct 31 01:24:07.490125 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.490060    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podStartSLOduration=1.4900426279999999 podStartE2EDuration="1.490042628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:07.488115855 +0000 UTC m=+1163.651259272" watchObservedRunningTime="2026-10-31 01:24:07.490042628 +0000 UTC m=+1163.653186041"
Oct 31 01:24:07.589869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:07.589836    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:08.011309 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:24:08.011158    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:24:08.591009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:08.590969    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:24:08.591473 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:08.591028    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:24:09.589636 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:09.589599    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:24:09.589940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:09.589679    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:24:10.589564 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:10.589529    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:24:10.589998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:10.589615    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:24:11.067619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.067584    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"]
Oct 31 01:24:11.072452 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.072422    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.094859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.094833    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.094995 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.094886    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095021    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095093 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095068    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095160    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095208 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095184    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095241 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095210    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j569g\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095271    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.095458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.095311    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.175365 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.175201    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"]
Oct 31 01:24:11.202071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202021    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202330 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202307    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202433 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202389    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202452    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202482    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202602 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202575    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202608    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202686    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j569g\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202728    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.202889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.202824    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.203230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.203204    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.203444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.203308    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.204201 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.204176    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.204413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.204393    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.206771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.206746    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.212735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.212713    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.269055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.269013    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.269244 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.269188    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j569g\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.388847 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.388764    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:11.590246 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.590213    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:24:11.590682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.590265    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:24:11.839617 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.839580    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"]
Oct 31 01:24:11.873640 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:24:11.873608    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e543206_2029_4141_b817_28b3fc200502.slice/crio-1b47ed528ed827a9a656e772d9b1038c758af3b78660b227bc3d8d1fbddc0887 WatchSource:0}: Error finding container 1b47ed528ed827a9a656e772d9b1038c758af3b78660b227bc3d8d1fbddc0887: Status 404 returned error can't find the container with id 1b47ed528ed827a9a656e772d9b1038c758af3b78660b227bc3d8d1fbddc0887
Oct 31 01:24:11.876369 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.876334    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:11.876471 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.876407    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:11.876471 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:11.876445    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:12.401731 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:12.401688    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" event={"ID":"3e543206-2029-4141-b817-28b3fc200502","Type":"ContainerStarted","Data":"03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c"}
Oct 31 01:24:12.401922 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:12.401759    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" event={"ID":"3e543206-2029-4141-b817-28b3fc200502","Type":"ContainerStarted","Data":"1b47ed528ed827a9a656e772d9b1038c758af3b78660b227bc3d8d1fbddc0887"}
Oct 31 01:24:12.527368 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:12.527317    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podStartSLOduration=1.527293905 podStartE2EDuration="1.527293905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:12.526311653 +0000 UTC m=+1168.689455070" watchObservedRunningTime="2026-10-31 01:24:12.527293905 +0000 UTC m=+1168.690437319"
Oct 31 01:24:12.598674 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:12.598267    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:24:12.598674 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:12.598362    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:24:13.389250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:13.389214    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:13.391006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:13.390982    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:24:13.391139 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:13.391037    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 01:24:13.590357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:13.590320    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 01:24:13.590538 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:13.590377    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 01:24:14.389366 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:14.389327    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:24:14.389832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:14.389396    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 01:24:14.678081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:14.678003    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:14.678601 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:14.678568    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:14.679471 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:14.679450    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:24:15.390007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:15.389972    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:24:15.390425 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:15.390026    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 01:24:16.390002 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:16.389970    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:24:16.390192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:16.390032    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 01:24:17.389278 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:17.389241    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:24:17.389491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:17.389321    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 01:24:18.389749 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.389712    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 31 01:24:18.390141 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.389770    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 31 01:24:18.868779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.868724    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"]
Oct 31 01:24:18.874084 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.874053    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909461 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909424    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909627 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909480    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2xb\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909627 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909595    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909637    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909728    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909752    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909772    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.909956 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.909929    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.910098 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.910074    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:18.927907 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.927880    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-baacee-openshift-default-dockercfg-pr4tt\""
Oct 31 01:24:18.987577 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:18.987541    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"]
Oct 31 01:24:19.011469 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011433    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011496    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011566    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011597    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2xb\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011644    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011693    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011729    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011759    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.011814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.011799    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.012380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.012020    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.012380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.012273    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.012529 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.012506    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.012587 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.012532    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.012970 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.012935    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.016226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.016182    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.016357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.016333    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.071222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.071185    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2xb\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.071453 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.071432    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.194915 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.194819    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:19.394159 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.394128    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:19.394584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.394456    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:19.395345 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.395328    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:24:19.680027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.679985    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"]
Oct 31 01:24:19.693125 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:24:19.693089    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856194b_a24b_4372_8d95_8dc546a9fa62.slice/crio-0c7dd448790a47328281f92ed842768cab16b9cd1847fa708d284ed3fe70f1cb WatchSource:0}: Error finding container 0c7dd448790a47328281f92ed842768cab16b9cd1847fa708d284ed3fe70f1cb: Status 404 returned error can't find the container with id 0c7dd448790a47328281f92ed842768cab16b9cd1847fa708d284ed3fe70f1cb
Oct 31 01:24:19.696392 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.696359    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:19.696492 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.696436    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:19.696492 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:19.696469    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:20.467631 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:20.467593    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" event={"ID":"8856194b-a24b-4372-8d95-8dc546a9fa62","Type":"ContainerStarted","Data":"a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34"}
Oct 31 01:24:20.468234 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:20.468211    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" event={"ID":"8856194b-a24b-4372-8d95-8dc546a9fa62","Type":"ContainerStarted","Data":"0c7dd448790a47328281f92ed842768cab16b9cd1847fa708d284ed3fe70f1cb"}
Oct 31 01:24:20.589451 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:20.589388    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podStartSLOduration=2.589372948 podStartE2EDuration="2.589372948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:20.58763002 +0000 UTC m=+1176.750773436" watchObservedRunningTime="2026-10-31 01:24:20.589372948 +0000 UTC m=+1176.752516362"
Oct 31 01:24:21.201611 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:21.200927    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:21.201611 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:21.201120    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:24:21.201611 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:21.201198    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 01:24:22.198677 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:22.198250    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:24:22.198677 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:22.198331    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 01:24:22.659745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:22.655765    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:23.199091 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:23.199054    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:24:23.199759 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:23.199723    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 01:24:24.205818 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:24.205780    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:24:24.219458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:24.219415    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 01:24:24.220078 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:24.218638    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:24:24.225813 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:24.224979    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-9mzcr" podUID="39645c3b-b2f4-448a-8683-c7b5b4ec1e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.559960 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:24.559911    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:24:24.560277 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:24.560240    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:25.195760 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:25.195724    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:24:25.195951 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:25.195779    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 01:24:26.195829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:26.195793    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 01:24:26.196359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:26.195854    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 01:24:27.636103 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:27.636068    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:27.636610 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:27.636521    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:27.637349 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:27.637318    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:24:33.997789 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:33.997752    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"]
Oct 31 01:24:34.001853 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.001832    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.102260 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.102228    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"]
Oct 31 01:24:34.111509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.111476    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.111771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.111528    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.111771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.111623    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.111897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.111791    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.111897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.111832    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.111897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.111858    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.112100 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.112079    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.112222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.112197    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.112519 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.112269    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdh8\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.213768 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.213730    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.213960 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.213858    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.213960 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.213885    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdh8\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.213960 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.213943    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.214128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.213983    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.214128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.214029    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.214128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.214115    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.214285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.214140    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.214285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.214163    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.214285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.214215    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.215006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.214800    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.215309 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.215282    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.215509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.215374    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.215836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.215812    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.216630 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.216607    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.217034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.217017    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.273638 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.273561    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.273820 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.273634    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdh8\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.316389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.316351    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:34.821269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.821082    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"]
Oct 31 01:24:34.838229 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.838183    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:34.838388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.838275    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:34.838388 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:34.838320    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:35.670109 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:35.669085    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" event={"ID":"753ea49d-2370-4c8f-bf9f-a32738d6369d","Type":"ContainerStarted","Data":"4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056"}
Oct 31 01:24:35.670109 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:35.669151    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" event={"ID":"753ea49d-2370-4c8f-bf9f-a32738d6369d","Type":"ContainerStarted","Data":"20924de8e37fe1a08dcc01708822889d3981f603add6f7ce0278b6183c66ce61"}
Oct 31 01:24:36.321705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:36.321669    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:36.322090 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:36.322064    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:36.322267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:36.322245    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:36.784676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:36.782747    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:37.026678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:37.023756    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:37.316999 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:37.316962    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:37.317197 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:37.317019    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:37.924235 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:24:37.924198    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:24:38.317668 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:38.317621    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:38.317867 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:38.317706    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:39.321112 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:39.321067    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:39.321613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:39.321180    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:40.317420 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:40.317382    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:40.317614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:40.317446    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:41.317077 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:41.317044    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:41.317493 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:41.317103    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:42.317571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:42.317433    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:24:42.318118 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:42.317596    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 01:24:43.322193 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:43.322156    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:43.322633 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:43.322529    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:43.323380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:43.323359    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:24:43.498640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:43.498585    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podStartSLOduration=10.498571584 podStartE2EDuration="10.498571584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:35.809468063 +0000 UTC m=+1191.972611474" watchObservedRunningTime="2026-10-31 01:24:43.498571584 +0000 UTC m=+1199.661714999"
Oct 31 01:24:50.362790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:50.362742    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 01:24:50.649872 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:50.649753    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:51.651355 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.651320    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"]
Oct 31 01:24:51.655978 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.655952    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.767541 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.767495    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.767730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.767582    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.767730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.767693    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88jm\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.767863 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.767843    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.767899 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.767882    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.767998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.767982    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.768102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.768087    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.768202 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.768166    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.768358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.768260    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.858692 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.858642    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"]
Oct 31 01:24:51.869854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.869823    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.869863    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.869944    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870147 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870027    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870147 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870061    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870147 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870135    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870183    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870240    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870275    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88jm\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870695 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870622    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.870940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.870919    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.871394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.871352    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.871477 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.871401    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.871477 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.871423    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.872867 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.872841    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.874034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.874012    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.928384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.928317    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88jm\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.928534 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.928488    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:51.975005 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:51.974974    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:52.414791 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.414761    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"]
Oct 31 01:24:52.415751 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:24:52.415725    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28711de_ee19_40ee_9e29_5049758bfd6e.slice/crio-fb35c8135c167d73a89150ca182bb76cbd655878ab8d4896f5386e4ffe5904f6 WatchSource:0}: Error finding container fb35c8135c167d73a89150ca182bb76cbd655878ab8d4896f5386e4ffe5904f6: Status 404 returned error can't find the container with id fb35c8135c167d73a89150ca182bb76cbd655878ab8d4896f5386e4ffe5904f6
Oct 31 01:24:52.418267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.418237    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:52.418347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.418301    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:52.418347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.418331    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:24:52.925104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.925062    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" event={"ID":"e28711de-ee19-40ee-9e29-5049758bfd6e","Type":"ContainerStarted","Data":"3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c"}
Oct 31 01:24:52.925527 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.925128    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" event={"ID":"e28711de-ee19-40ee-9e29-5049758bfd6e","Type":"ContainerStarted","Data":"fb35c8135c167d73a89150ca182bb76cbd655878ab8d4896f5386e4ffe5904f6"}
Oct 31 01:24:52.976026 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:52.975990    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:53.096285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:53.096228    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podStartSLOduration=2.096211311 podStartE2EDuration="2.096211311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:53.094229877 +0000 UTC m=+1209.257373329" watchObservedRunningTime="2026-10-31 01:24:53.096211311 +0000 UTC m=+1209.259354726"
Oct 31 01:24:53.977319 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:53.977285    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 01:24:53.977784 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:53.977333    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 01:24:54.976282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:54.976249    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 01:24:54.976495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:54.976313    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 01:24:55.975514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:55.975479    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 01:24:55.975930 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:55.975531    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 01:24:56.975291 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:56.975254    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 01:24:56.975484 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:56.975317    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 01:24:57.975568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:57.975524    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 01:24:57.976095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:57.975593    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 01:24:58.975842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:58.975798    5718 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 31 01:24:58.976272 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:58.975875    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 31 01:24:59.979808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:59.979778    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:59.980199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:59.980010    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:24:59.980668 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:24:59.980639    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 01:25:04.362674 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:04.353775    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 31 01:25:04.398293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:04.397844    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.890680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:04.878090    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:05.141752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:05.140814    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:25:05.141752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:05.140892    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:07.924616 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:25:07.924580    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:25:17.666560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.666517    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"]
Oct 31 01:25:17.671452 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.671433    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.726984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.726945    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-00e840-openshift-default-dockercfg-2nfqp\""
Oct 31 01:25:17.755907 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.755873    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.755907 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.755915    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.755950    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.756050    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hmn\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.756092    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756309 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.756180    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756392 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.756366    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756461 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.756409    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.756551 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.756527    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.781643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.781608    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"]
Oct 31 01:25:17.857522 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857481    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.857522 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857528    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hmn\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.857833 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857569    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.857833 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857705    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.857833 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857820    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.857990 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857848    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.857990 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.857962    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858030    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858059    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858157    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858200    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858470 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858443    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858556    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.858647 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.858573    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.860271 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.860251    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.860380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.860362    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.915485 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.915452    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hmn\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.915645 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.915491    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:17.986479 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:17.986393    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:18.415194 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:18.415168    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"]
Oct 31 01:25:18.415809 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:25:18.415771    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bfe2b5_f59c_4f4b_9295_eadfcc164548.slice/crio-388cd43e5608feffb7f22fa330f5c14b1b98185923e5ef584b4333567eff741c WatchSource:0}: Error finding container 388cd43e5608feffb7f22fa330f5c14b1b98185923e5ef584b4333567eff741c: Status 404 returned error can't find the container with id 388cd43e5608feffb7f22fa330f5c14b1b98185923e5ef584b4333567eff741c
Oct 31 01:25:18.418423 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:18.418388    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:25:18.418519 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:18.418458    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:25:18.418519 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:18.418487    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:25:19.249270 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.249232    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" event={"ID":"a9bfe2b5-f59c-4f4b-9295-eadfcc164548","Type":"ContainerStarted","Data":"5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84"}
Oct 31 01:25:19.249708 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.249294    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" event={"ID":"a9bfe2b5-f59c-4f4b-9295-eadfcc164548","Type":"ContainerStarted","Data":"388cd43e5608feffb7f22fa330f5c14b1b98185923e5ef584b4333567eff741c"}
Oct 31 01:25:19.392072 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.392040    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"]
Oct 31 01:25:19.396086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.396065    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.467810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.467743    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podStartSLOduration=2.46772591 podStartE2EDuration="2.46772591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:19.422157077 +0000 UTC m=+1235.585300495" watchObservedRunningTime="2026-10-31 01:25:19.46772591 +0000 UTC m=+1235.630869387"
Oct 31 01:25:19.485487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.485445    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.485693 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.485588    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.485777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.485739    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.485845 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.485778    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.485901 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.485877    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.485949 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.485908    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.486115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.486094    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.486183 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.486144    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.486242 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.486231    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5f4g\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.533798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.533722    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"]
Oct 31 01:25:19.587949 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.587913    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.587956    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588002    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588140 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588023    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588140    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588180    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588238    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5f4g\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588300    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588500 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588380    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588500 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588406    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588514    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.588987 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.588958    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.589269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.589242    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.589528 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.589508    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.591722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.591696    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.591989 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.591944    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.668712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.668673    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.669013 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.668990    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5f4g\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.713237 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.713205    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:19.986507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.986477    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:19.988481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.988452    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:19.988641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:19.988511    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:20.178776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.178741    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"]
Oct 31 01:25:20.183545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.183515    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:25:20.183640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.183579    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:25:20.183640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.183607    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:25:20.271908 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.271867    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" event={"ID":"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a","Type":"ContainerStarted","Data":"b88dab5f428a25dc78899c7839f1d030dbb7dbb191c4f91b0c2430b906e163f0"}
Oct 31 01:25:20.988932 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.988893    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:20.989150 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:20.988958    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:21.315776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.315736    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" event={"ID":"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a","Type":"ContainerStarted","Data":"2f2769ddd99873aef644d2a34620e5a7b00c18eb4e8260a592ffbc929243f1e5"}
Oct 31 01:25:21.560750 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.560676    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podStartSLOduration=2.5606323189999998 podStartE2EDuration="2.560632319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:21.490120431 +0000 UTC m=+1237.653263846" watchObservedRunningTime="2026-10-31 01:25:21.560632319 +0000 UTC m=+1237.723775735"
Oct 31 01:25:21.713570 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.713486    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:21.715154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.715132    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:21.715236 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.715183    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:21.999676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.998115    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:21.999676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:21.998177    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:22.715678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:22.715108    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:22.715678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:22.715172    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:22.989229 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:22.989129    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:22.989539 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:22.989488    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:23.713705 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:23.713670    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:23.714031 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:23.713982    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:23.886681 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:23.884784    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.005945 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.005900    5718 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.027678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.009749    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:24.027678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.009801    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:24.027678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.026738    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.258933 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.239131    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:25:24.364739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.362779    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.399825 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.399782    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.541692 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.536763    5718 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.663678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.661755    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.723679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.723081    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:24.723679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.723147    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:24.783678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.781811    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.889737 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.887759    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.889737 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.888209    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:25:24.990679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.990253    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:24.990679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:24.990334    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:25.031796 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.028757    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:25.031796 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.029206    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:25:25.241738 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.241175    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 01:25:25.241738 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.241599    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:25:25.456680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.400290    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 31 01:25:25.460729 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.419244    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:25.595679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.568447    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:25:25.595679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.569830    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:25:25.640444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.639417    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 01:25:25.640444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.639478    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 01:25:25.704675 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.701133    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 01:25:25.719717 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.719280    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:25.719717 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.719363    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:25.720552 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.720525    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 01:25:25.720898 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.720874    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 01:25:25.987840 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.987751    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 01:25:25.988976 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:25.988755    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 01:25:26.714245 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:26.714212    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:26.714437 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:26.714279    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:27.112535 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:27.112499    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:27.113106 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:27.113053    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:27.114108 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:27.114057    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:25:27.713490 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:27.713458    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:25:27.713699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:27.713521    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:25:28.858643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:28.858611    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:28.859272 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:28.859192    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:28.860329 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:28.860306    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:25:36.017812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.017777    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"]
Oct 31 01:25:36.021413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.021390    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.077110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.077069    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-18bf762c-serving-cert\""
Oct 31 01:25:36.130688 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.130626    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"]
Oct 31 01:25:36.171437 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.171392    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.171613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.171534    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.171725 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.171706    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwzg\" (UniqueName: \"kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.171805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.171791    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.275201 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.275103    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.281917 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.281876    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.282128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.281993    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.282204 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.282141    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwzg\" (UniqueName: \"kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.282339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.282314    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.285554 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.285525    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.286697 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.286673    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.341175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.341146    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwzg\" (UniqueName: \"kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg\") pod \"maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:36.634507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:36.634415    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:37.015226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:37.015194    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"]
Oct 31 01:25:37.016974 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:25:37.016948    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84517f1_e361_44ed_a104_88645e37ff3e.slice/crio-991642c58d7d245468d68b63cd5c0ed071475b12b90e0d894550751b56b1a297 WatchSource:0}: Error finding container 991642c58d7d245468d68b63cd5c0ed071475b12b90e0d894550751b56b1a297: Status 404 returned error can't find the container with id 991642c58d7d245468d68b63cd5c0ed071475b12b90e0d894550751b56b1a297
Oct 31 01:25:37.018515 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:37.018496    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:25:37.633354 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:37.633319    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w" event={"ID":"c84517f1-e361-44ed-a104-88645e37ff3e","Type":"ContainerStarted","Data":"991642c58d7d245468d68b63cd5c0ed071475b12b90e0d894550751b56b1a297"}
Oct 31 01:25:37.924191 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:25:37.924104    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:25:38.642362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:38.642322    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w" event={"ID":"c84517f1-e361-44ed-a104-88645e37ff3e","Type":"ContainerStarted","Data":"eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd"}
Oct 31 01:25:38.642864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:38.642430    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:38.765644 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:38.765586    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w" podStartSLOduration=2.758843955 podStartE2EDuration="3.76557194s" totalImagesPullingTime="1.006727985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:38.764320934 +0000 UTC m=+1254.927464350" watchObservedRunningTime="2026-10-31 01:25:38.76557194 +0000 UTC m=+1254.928715355"
Oct 31 01:25:44.655348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:44.655316    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:25:55.723832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:25:55.723705    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:07.935217 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:26:07.935138    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:26:09.587905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:26:09.587816    5718 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-7f274 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:09.588636 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:26:09.588588    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:37.924901 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:26:37.924850    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:27:07.925161 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:27:07.925122    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:27:11.412483 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.412454    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"]
Oct 31 01:27:11.416717 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.416689    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.527063 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.527032    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"]
Oct 31 01:27:11.575735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.575688    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.575905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.575753    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.575905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.575866    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.575985 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.575910    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.575985 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.575931    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.576067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.576053    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.576102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.576085    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmll\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.576179 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.576161    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.576238 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.576221    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677239 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677151    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677239 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677232    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677279    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677410    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677539 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677481    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677539 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677514    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677633 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677613    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677638    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677702    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmll\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677710    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677786    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.677897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.677848    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.678132 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.678105    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.678362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.678304    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.679949 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.679930    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.680082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.680064    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.736559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.736517    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:11.736793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:11.736583    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmll\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:12.030940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:12.030903    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:12.463938 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:27:12.463910    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f26f55_dc50_4e43_8c5c_da1a374a2c56.slice/crio-f55911d959d72165a269ca60d50b1a5484e1ba5f22f4feed5d0113774db2cc25 WatchSource:0}: Error finding container f55911d959d72165a269ca60d50b1a5484e1ba5f22f4feed5d0113774db2cc25: Status 404 returned error can't find the container with id f55911d959d72165a269ca60d50b1a5484e1ba5f22f4feed5d0113774db2cc25
Oct 31 01:27:12.464291 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:12.464025    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"]
Oct 31 01:27:12.466583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:12.466550    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:12.466709 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:12.466624    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:12.466709 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:12.466701    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:13.307823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:13.307786    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" event={"ID":"15f26f55-dc50-4e43-8c5c-da1a374a2c56","Type":"ContainerStarted","Data":"0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7"}
Oct 31 01:27:13.308031 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:13.307848    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" event={"ID":"15f26f55-dc50-4e43-8c5c-da1a374a2c56","Type":"ContainerStarted","Data":"f55911d959d72165a269ca60d50b1a5484e1ba5f22f4feed5d0113774db2cc25"}
Oct 31 01:27:13.477484 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:13.477429    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podStartSLOduration=2.477416081 podStartE2EDuration="2.477416081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:13.476502321 +0000 UTC m=+1349.639645733" watchObservedRunningTime="2026-10-31 01:27:13.477416081 +0000 UTC m=+1349.640559496"
Oct 31 01:27:14.031998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:14.031958    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:14.033451 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:14.033425    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:27:14.033583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:14.033483    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:27:15.031757 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:15.031722    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:27:15.032184 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:15.031772    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:27:16.032127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:16.032092    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:27:16.032533 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:16.032146    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:27:17.031803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:17.031766    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:27:17.031979 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:17.031839    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:27:18.031426 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:18.031385    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:27:18.031925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:18.031457    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:27:19.032206 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:19.032170    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 01:27:19.032678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:19.032273    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 01:27:20.036103 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:20.036064    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:20.036562 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:20.036470    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:20.037190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:20.037166    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:27:37.924990 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:27:37.924951    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:27:40.599097 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.599053    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"]
Oct 31 01:27:40.603343 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.603324    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.661029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.661000    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"]
Oct 31 01:27:40.698272 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.698229    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.698455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.698292    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.698455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.698414    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.698566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.698455    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.698622 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.698581    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxl4\" (UniqueName: \"kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.698769 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.698752    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.707299 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.707274    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"openshift-service-ca.crt\""
Oct 31 01:27:40.707456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.707321    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"default-dockercfg-28mfd\""
Oct 31 01:27:40.707456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.707375    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"kube-root-ca.crt\""
Oct 31 01:27:40.707537 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.707471    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"distinct-2-w4-8f6b5b-kserve-self-signed-certs\""
Oct 31 01:27:40.800395 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.800346    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.800598 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.800422    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.800598 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.800484    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxl4\" (UniqueName: \"kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.800801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.800623    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.800801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.800775    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.800921 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.800871    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.801248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.801224    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.801334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.801230    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.801582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.801566    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.803698 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.803672    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.803910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.803891    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.861697 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.861597    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxl4\" (UniqueName: \"kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4\") pod \"distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:40.916968 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:40.916918    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:41.356827 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:41.356792    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"]
Oct 31 01:27:41.357526 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:27:41.357490    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7771fbf3_2ed2_477e_890d_f9c2b7a6b5a9.slice/crio-819f67a323af3a86b78b53a5c2d2a63fb0876b192dd7154d2f9a1dcb482da7d1 WatchSource:0}: Error finding container 819f67a323af3a86b78b53a5c2d2a63fb0876b192dd7154d2f9a1dcb482da7d1: Status 404 returned error can't find the container with id 819f67a323af3a86b78b53a5c2d2a63fb0876b192dd7154d2f9a1dcb482da7d1
Oct 31 01:27:41.497017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:41.496977    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc" event={"ID":"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9","Type":"ContainerStarted","Data":"0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617"}
Oct 31 01:27:41.497017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:41.497023    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc" event={"ID":"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9","Type":"ContainerStarted","Data":"819f67a323af3a86b78b53a5c2d2a63fb0876b192dd7154d2f9a1dcb482da7d1"}
Oct 31 01:27:41.618667 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:41.618553    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc" podStartSLOduration=1.618538532 podStartE2EDuration="1.618538532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:41.616803439 +0000 UTC m=+1377.779946855" watchObservedRunningTime="2026-10-31 01:27:41.618538532 +0000 UTC m=+1377.781681947"
Oct 31 01:27:41.917950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:41.917864    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:48.831798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.831763    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"]
Oct 31 01:27:48.835810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.835792    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.849971 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.849939    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850017    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850053    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850079    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850147    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850180    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850321 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850268    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7cgb\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850414    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.850491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.850478    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.859734 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.859695    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"]
Oct 31 01:27:48.863829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.863807    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.909343 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.909299    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"]
Oct 31 01:27:48.952290 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952257    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.952567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952328    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.952567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952366    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.952567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952400    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.952567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952447    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.952567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952473    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.952567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952537    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952609    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952682    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952723    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952755    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952795    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952855    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7cgb\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952900    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952926    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpqbh\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.953007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.952975    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.953466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.953012    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.953466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.953038    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.953466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.953082    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.953466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.953105    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:48.953466 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.953328    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.954857 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.953999    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.954968 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.954892    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.966188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.962278    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"]
Oct 31 01:27:48.966188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.965610    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:48.969468 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:48.969440    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:49.017384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.017312    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7cgb\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:49.017715 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.017684    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:49.054051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054016    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054064    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054092    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054159    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054254    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpqbh\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054286    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054328    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054408 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054377    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054407    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054561    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.054751 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054637    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.055008 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.054977    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.055117 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.055038    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.055407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.055340    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.056957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.056933    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.057053 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.056962    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.116817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.116735    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.117111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.117086    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpqbh\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.151007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.150975    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:49.177444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.177412    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:49.584283 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.584254    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"]
Oct 31 01:27:49.585078 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:27:49.585049    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838b972f_e0d5_42f7_b962_69c3cbd6de86.slice/crio-e1af94b622208938be3a6dd7117965ca650e09d071b1d4abcd0c0b88295305b9 WatchSource:0}: Error finding container e1af94b622208938be3a6dd7117965ca650e09d071b1d4abcd0c0b88295305b9: Status 404 returned error can't find the container with id e1af94b622208938be3a6dd7117965ca650e09d071b1d4abcd0c0b88295305b9
Oct 31 01:27:49.587467 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.587442    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"]
Oct 31 01:27:49.587868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.587837    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:49.587927 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.587901    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:49.587966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.587933    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:49.620909 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:27:49.620872    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621b8137_c8de_42aa_8637_132c54e7e5cf.slice/crio-c886e7ee95c3b718e5a783ae9ba463ccf91633f19a65a1aaec01b3b16f36184b WatchSource:0}: Error finding container c886e7ee95c3b718e5a783ae9ba463ccf91633f19a65a1aaec01b3b16f36184b: Status 404 returned error can't find the container with id c886e7ee95c3b718e5a783ae9ba463ccf91633f19a65a1aaec01b3b16f36184b
Oct 31 01:27:49.623562 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.623527    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:49.623688 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.623616    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:49.623688 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:49.623679    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:27:50.602273 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.602237    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" event={"ID":"621b8137-c8de-42aa-8637-132c54e7e5cf","Type":"ContainerStarted","Data":"460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a"}
Oct 31 01:27:50.602735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.602291    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" event={"ID":"621b8137-c8de-42aa-8637-132c54e7e5cf","Type":"ContainerStarted","Data":"c886e7ee95c3b718e5a783ae9ba463ccf91633f19a65a1aaec01b3b16f36184b"}
Oct 31 01:27:50.605582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.605555    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" event={"ID":"838b972f-e0d5-42f7-b962-69c3cbd6de86","Type":"ContainerStarted","Data":"d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7"}
Oct 31 01:27:50.605701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.605600    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" event={"ID":"838b972f-e0d5-42f7-b962-69c3cbd6de86","Type":"ContainerStarted","Data":"e1af94b622208938be3a6dd7117965ca650e09d071b1d4abcd0c0b88295305b9"}
Oct 31 01:27:50.840216 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.840157    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podStartSLOduration=2.840127129 podStartE2EDuration="2.840127129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:50.837485643 +0000 UTC m=+1387.000629061" watchObservedRunningTime="2026-10-31 01:27:50.840127129 +0000 UTC m=+1387.003270545"
Oct 31 01:27:50.918017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.917918    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:50.930543 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:50.930517    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:51.126231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.125533    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podStartSLOduration=3.125516881 podStartE2EDuration="3.125516881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:50.997064107 +0000 UTC m=+1387.160207523" watchObservedRunningTime="2026-10-31 01:27:51.125516881 +0000 UTC m=+1387.288660295"
Oct 31 01:27:51.154531 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.154493    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:51.154925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.154901    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:27:51.155104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.155079    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 01:27:51.186536 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.186445    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:51.186908 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.186884    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:27:51.187090 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.187063    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:27:51.624639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:51.624612    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 01:27:52.172679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:52.161042    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:27:52.172679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:52.161104    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 01:27:52.191701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:52.186077    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:27:52.191701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:52.186158    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:27:53.153691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:53.153118    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:27:53.153691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:53.153182    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 01:27:53.185680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:53.181778    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:27:53.185680 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:53.181846    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:27:54.162685 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:54.162087    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:27:54.162685 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:54.162157    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 01:27:54.178068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:54.178031    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:27:54.179473 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:54.179439    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:27:55.185684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:55.181341    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:27:55.185684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:55.181417    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 01:27:55.198676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:55.193690    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:27:55.198676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:55.193765    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:27:56.151468 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:56.151434    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 01:27:56.151714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:56.151492    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 01:27:56.178067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:56.178024    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:27:56.178238 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:56.178098    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:27:57.247090 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:57.247062    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:57.247555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:57.247527    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:57.249314 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:57.249282    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:27:57.281216 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:57.281189    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:57.281940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:57.281917    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:27:57.283136 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:27:57.283115    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:28:07.924251 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:28:07.924210    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:28:32.797878 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.797849    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"]
Oct 31 01:28:32.801780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.801760    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.859480 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.859442    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"]
Oct 31 01:28:32.882158 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.882122    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24tp2\" (UniqueName: \"kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.882373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.882256    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.882436 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.882398    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.882501 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.882484    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.905455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.905422    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-01cbd8-serving-cert\""
Oct 31 01:28:32.984311 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.984279    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.984535 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.984344    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.984535 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.984466    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tp2\" (UniqueName: \"kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.984535 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:28:32.984494    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-serving-cert: secret "maas-api-e2e-api-a-01cbd8-serving-cert" not found
Oct 31 01:28:32.984742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.984553    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.984742 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:28:32.984572    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls podName:51fa7641-31ba-411c-b5ed-80f6e8996227 nodeName:}" failed. No retries permitted until 2026-10-31 01:28:33.484549413 +0000 UTC m=+1429.647692824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls") pod "maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" (UID: "51fa7641-31ba-411c-b5ed-80f6e8996227") : secret "maas-api-e2e-api-a-01cbd8-serving-cert" not found
Oct 31 01:28:32.985369 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.985348    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:32.987008 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:32.986985    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:33.042329 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:33.042297    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tp2\" (UniqueName: \"kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:33.490836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:33.490790    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:33.493749 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:33.493717    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\") pod \"maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:33.717788 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:33.717750    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:34.098949 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:28:34.098911    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa7641_31ba_411c_b5ed_80f6e8996227.slice/crio-a1fbf16911a146d8180d484fe4425d0f81ea3c4b6187aea3573ed160c447305e WatchSource:0}: Error finding container a1fbf16911a146d8180d484fe4425d0f81ea3c4b6187aea3573ed160c447305e: Status 404 returned error can't find the container with id a1fbf16911a146d8180d484fe4425d0f81ea3c4b6187aea3573ed160c447305e
Oct 31 01:28:34.131916 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:34.131884    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"]
Oct 31 01:28:35.031578 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:35.031543    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" event={"ID":"51fa7641-31ba-411c-b5ed-80f6e8996227","Type":"ContainerStarted","Data":"a1fbf16911a146d8180d484fe4425d0f81ea3c4b6187aea3573ed160c447305e"}
Oct 31 01:28:36.042035 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:36.041985    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" event={"ID":"51fa7641-31ba-411c-b5ed-80f6e8996227","Type":"ContainerStarted","Data":"1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8"}
Oct 31 01:28:36.042576 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:36.042091    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:28:36.164030 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:36.163965    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" podStartSLOduration=3.272745101 podStartE2EDuration="4.163950889s" totalImagesPullingTime="891.205788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:36.16199403 +0000 UTC m=+1432.325137472" watchObservedRunningTime="2026-10-31 01:28:36.163950889 +0000 UTC m=+1432.327094303"
Oct 31 01:28:37.924761 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:28:37.924716    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:28:42.055166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:28:42.055135    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:29:07.925216 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:29:07.925173    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:29:27.127037 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.127001    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"]
Oct 31 01:29:27.130971 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.130951    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.183851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.183807    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:29:27.183851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.183807    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:29:27.183851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.183807    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:29:27.234589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.234553    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-85vx5\""
Oct 31 01:29:27.238265 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.238232    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"]
Oct 31 01:29:27.267989 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.267958    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.268199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.268058    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.268199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.268108    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.268330 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.268221    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.268402 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.268386    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.268458 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.268414    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws262\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.268559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.268538    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369525    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369566    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws262\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369840 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369621    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369840 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369710    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369840 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369796    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369993 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369845    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.369993 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.369917    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.370910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.370877    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.372185 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.372157    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.372474 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.372452    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.372731 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.372708    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.372731 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.372721    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.427302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.427134    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.427520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.427452    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws262\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f4lr4\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.444224 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.444188    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:27.873438 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:27.873409    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"]
Oct 31 01:29:27.874237 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:29:27.874201    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b03c8b3_2538_4079_bfb7_49eeb795379a.slice/crio-4211f4b5c20e2afd9eeb8e16a2adea25fa2ed9091a179556d66457536289e7f7 WatchSource:0}: Error finding container 4211f4b5c20e2afd9eeb8e16a2adea25fa2ed9091a179556d66457536289e7f7: Status 404 returned error can't find the container with id 4211f4b5c20e2afd9eeb8e16a2adea25fa2ed9091a179556d66457536289e7f7
Oct 31 01:29:28.393833 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:28.393797    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" event={"ID":"5b03c8b3-2538-4079-bfb7-49eeb795379a","Type":"ContainerStarted","Data":"4211f4b5c20e2afd9eeb8e16a2adea25fa2ed9091a179556d66457536289e7f7"}
Oct 31 01:29:31.662848 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:31.662814    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:29:31.663214 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:31.662884    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:29:32.423959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:32.423918    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" event={"ID":"5b03c8b3-2538-4079-bfb7-49eeb795379a","Type":"ContainerStarted","Data":"9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353"}
Oct 31 01:29:32.424155 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:32.424059    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:32.425852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:32.425828    5718 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-f4lr4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:29:32.425972 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:32.425878    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" podUID="5b03c8b3-2538-4079-bfb7-49eeb795379a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:29:32.640286 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:32.640225    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" podStartSLOduration=1.855154895 podStartE2EDuration="5.64021106s" totalImagesPullingTime="3.785056165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:32.638044262 +0000 UTC m=+1488.801187679" watchObservedRunningTime="2026-10-31 01:29:32.64021106 +0000 UTC m=+1488.803354475"
Oct 31 01:29:33.431448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:29:33.431410    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:29:37.924979 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:29:37.924934    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:30:00.182419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.182383    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"]
Oct 31 01:30:00.186732 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.186632    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"]
Oct 31 01:30:00.187188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.186806    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.190804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.190780    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.240130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.240083    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:30:00.295053 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.295023    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"]
Oct 31 01:30:00.298085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.298051    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"]
Oct 31 01:30:00.329843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.329806    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.330025 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.329928    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-bp97r\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.330159 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.330137    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ds7s\" (UniqueName: \"kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s\") pod \"maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.330196 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.330169    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjz8p\" (UniqueName: \"kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p\") pod \"maas-api-key-cleanup-29890170-bp97r\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.431684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.431613    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds7s\" (UniqueName: \"kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s\") pod \"maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.431684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.431672    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjz8p\" (UniqueName: \"kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p\") pod \"maas-api-key-cleanup-29890170-bp97r\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.431953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.431733    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.431953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.431802    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-bp97r\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.432411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.432392    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.432477 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.432427    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-bp97r\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.489324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.489294    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ds7s\" (UniqueName: \"kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s\") pod \"maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.489608 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.489587    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjz8p\" (UniqueName: \"kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p\") pod \"maas-api-key-cleanup-29890170-bp97r\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.505700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.505640    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:00.512193 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.512171    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:00.893195 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:30:00.893165    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb47329_2b5d_4cc1_9643_2f51d2cbaddb.slice/crio-dd11c5bacea7d3fa52cf9479f56205895fb8e96f89de0a944b9ff6fc4360be4c WatchSource:0}: Error finding container dd11c5bacea7d3fa52cf9479f56205895fb8e96f89de0a944b9ff6fc4360be4c: Status 404 returned error can't find the container with id dd11c5bacea7d3fa52cf9479f56205895fb8e96f89de0a944b9ff6fc4360be4c
Oct 31 01:30:00.894092 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.894052    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"]
Oct 31 01:30:00.945677 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:00.945637    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"]
Oct 31 01:30:00.948080 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:30:00.948051    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode884c4f9_2c44_4b57_8a17_59af1ac6f614.slice/crio-9b05970a1769deb1cf8c3fa3a6b9b991db00d8c65ba89bc2dad19eaf456ca0ed WatchSource:0}: Error finding container 9b05970a1769deb1cf8c3fa3a6b9b991db00d8c65ba89bc2dad19eaf456ca0ed: Status 404 returned error can't find the container with id 9b05970a1769deb1cf8c3fa3a6b9b991db00d8c65ba89bc2dad19eaf456ca0ed
Oct 31 01:30:01.616251 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:01.616210    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerStarted","Data":"a528bdcd62de7af71ae401830164b3dd466d546ae41ae5b4a5f00e6336bdd6c0"}
Oct 31 01:30:01.616803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:01.616273    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerStarted","Data":"9b05970a1769deb1cf8c3fa3a6b9b991db00d8c65ba89bc2dad19eaf456ca0ed"}
Oct 31 01:30:01.618094 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:01.618065    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" event={"ID":"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb","Type":"ContainerStarted","Data":"0aea05fe5fd20073c9bda476a4bbe705eef0308f3c4202dac193dec4134f5dc2"}
Oct 31 01:30:01.618219 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:01.618110    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" event={"ID":"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb","Type":"ContainerStarted","Data":"dd11c5bacea7d3fa52cf9479f56205895fb8e96f89de0a944b9ff6fc4360be4c"}
Oct 31 01:30:01.870156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:01.870066    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" podStartSLOduration=1.870050747 podStartE2EDuration="1.870050747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:01.869231198 +0000 UTC m=+1518.032374613" watchObservedRunningTime="2026-10-31 01:30:01.870050747 +0000 UTC m=+1518.033194161"
Oct 31 01:30:01.871765 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:01.871727    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" podStartSLOduration=1.871712883 podStartE2EDuration="1.871712883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:01.746825281 +0000 UTC m=+1517.909968695" watchObservedRunningTime="2026-10-31 01:30:01.871712883 +0000 UTC m=+1518.034856297"
Oct 31 01:30:02.627060 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:02.627019    5718 generic.go:356] "Generic (PLEG): container finished" podID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerID="a528bdcd62de7af71ae401830164b3dd466d546ae41ae5b4a5f00e6336bdd6c0" exitCode=60
Oct 31 01:30:02.627559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:02.627113    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerDied","Data":"a528bdcd62de7af71ae401830164b3dd466d546ae41ae5b4a5f00e6336bdd6c0"}
Oct 31 01:30:02.627559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:02.627458    5718 scope.go:122] "RemoveContainer" containerID="a528bdcd62de7af71ae401830164b3dd466d546ae41ae5b4a5f00e6336bdd6c0"
Oct 31 01:30:02.628843 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:02.628819    5718 generic.go:356] "Generic (PLEG): container finished" podID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerID="0aea05fe5fd20073c9bda476a4bbe705eef0308f3c4202dac193dec4134f5dc2" exitCode=60
Oct 31 01:30:02.628934 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:02.628919    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" event={"ID":"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb","Type":"ContainerDied","Data":"0aea05fe5fd20073c9bda476a4bbe705eef0308f3c4202dac193dec4134f5dc2"}
Oct 31 01:30:02.629177 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:02.629161    5718 scope.go:122] "RemoveContainer" containerID="0aea05fe5fd20073c9bda476a4bbe705eef0308f3c4202dac193dec4134f5dc2"
Oct 31 01:30:03.637169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.637137    5718 generic.go:356] "Generic (PLEG): container finished" podID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerID="3f69db07980f8b443afc24138959dbf475d851e94efad13e1d9f5b2fbb3f15ee" exitCode=60
Oct 31 01:30:03.637586 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.637225    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerDied","Data":"3f69db07980f8b443afc24138959dbf475d851e94efad13e1d9f5b2fbb3f15ee"}
Oct 31 01:30:03.637586 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.637288    5718 scope.go:122] "RemoveContainer" containerID="a528bdcd62de7af71ae401830164b3dd466d546ae41ae5b4a5f00e6336bdd6c0"
Oct 31 01:30:03.637758 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.637589    5718 scope.go:122] "RemoveContainer" containerID="3f69db07980f8b443afc24138959dbf475d851e94efad13e1d9f5b2fbb3f15ee"
Oct 31 01:30:03.638017 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:03.637998    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45_odh-ai-gateway-infra(e884c4f9-2c44-4b57-8a17-59af1ac6f614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614"
Oct 31 01:30:03.639519 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.639490    5718 generic.go:356] "Generic (PLEG): container finished" podID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerID="98f1648be43e6963d1e22f4b0475767f64d1fc852697c8bafe18407e6ead4ee4" exitCode=60
Oct 31 01:30:03.639625 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.639589    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" event={"ID":"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb","Type":"ContainerDied","Data":"98f1648be43e6963d1e22f4b0475767f64d1fc852697c8bafe18407e6ead4ee4"}
Oct 31 01:30:03.640146 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.640130    5718 scope.go:122] "RemoveContainer" containerID="98f1648be43e6963d1e22f4b0475767f64d1fc852697c8bafe18407e6ead4ee4"
Oct 31 01:30:03.640506 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:03.640489    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-bp97r_odh-ai-gateway-infra(8cb47329-2b5d-4cc1-9643-2f51d2cbaddb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb"
Oct 31 01:30:03.689710 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:03.689683    5718 scope.go:122] "RemoveContainer" containerID="0aea05fe5fd20073c9bda476a4bbe705eef0308f3c4202dac193dec4134f5dc2"
Oct 31 01:30:04.650673 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:04.648593    5718 scope.go:122] "RemoveContainer" containerID="3f69db07980f8b443afc24138959dbf475d851e94efad13e1d9f5b2fbb3f15ee"
Oct 31 01:30:04.650673 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:04.649427    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45_odh-ai-gateway-infra(e884c4f9-2c44-4b57-8a17-59af1ac6f614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614"
Oct 31 01:30:04.653023 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:04.652987    5718 scope.go:122] "RemoveContainer" containerID="98f1648be43e6963d1e22f4b0475767f64d1fc852697c8bafe18407e6ead4ee4"
Oct 31 01:30:04.653396 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:04.653368    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-bp97r_odh-ai-gateway-infra(8cb47329-2b5d-4cc1-9643-2f51d2cbaddb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb"
Oct 31 01:30:07.925345 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:07.925306    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:30:12.708981 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:12.708940    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"]
Oct 31 01:30:12.709427 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:12.709216    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w" podUID="c84517f1-e361-44ed-a104-88645e37ff3e" containerName="maas-api" containerID="cri-o://eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd" gracePeriod=30
Oct 31 01:30:13.033923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.033895    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:30:13.135088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.134119    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls\") pod \"c84517f1-e361-44ed-a104-88645e37ff3e\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") "
Oct 31 01:30:13.135088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.134231    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg\" (UniqueName: \"kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg\") pod \"c84517f1-e361-44ed-a104-88645e37ff3e\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") "
Oct 31 01:30:13.135088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.134295    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca\") pod \"c84517f1-e361-44ed-a104-88645e37ff3e\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") "
Oct 31 01:30:13.135088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.134360    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs\") pod \"c84517f1-e361-44ed-a104-88645e37ff3e\" (UID: \"c84517f1-e361-44ed-a104-88645e37ff3e\") "
Oct 31 01:30:13.136079 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.135973    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca" pod "c84517f1-e361-44ed-a104-88645e37ff3e" (UID: "c84517f1-e361-44ed-a104-88645e37ff3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:13.138499 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.138402    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs" pod "c84517f1-e361-44ed-a104-88645e37ff3e" (UID: "c84517f1-e361-44ed-a104-88645e37ff3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:30:13.152722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.150925    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls" pod "c84517f1-e361-44ed-a104-88645e37ff3e" (UID: "c84517f1-e361-44ed-a104-88645e37ff3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:30:13.152722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.150949    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg" pod "c84517f1-e361-44ed-a104-88645e37ff3e" (UID: "c84517f1-e361-44ed-a104-88645e37ff3e"). InnerVolumeSpecName "kube-api-access-pvwzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:13.237162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.237127    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:13.237162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.237160    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvwzg\" (UniqueName: \"kubernetes.io/projected/c84517f1-e361-44ed-a104-88645e37ff3e-kube-api-access-pvwzg\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:13.237162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.237170    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84517f1-e361-44ed-a104-88645e37ff3e-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:13.237407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.237181    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84517f1-e361-44ed-a104-88645e37ff3e-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:13.740635 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.740596    5718 generic.go:356] "Generic (PLEG): container finished" podID="c84517f1-e361-44ed-a104-88645e37ff3e" containerID="eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd" exitCode=0
Oct 31 01:30:13.740635 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.740677    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"
Oct 31 01:30:13.741289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.740698    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w" event={"ID":"c84517f1-e361-44ed-a104-88645e37ff3e","Type":"ContainerDied","Data":"eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd"}
Oct 31 01:30:13.741289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.740766    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w" event={"ID":"c84517f1-e361-44ed-a104-88645e37ff3e","Type":"ContainerDied","Data":"991642c58d7d245468d68b63cd5c0ed071475b12b90e0d894550751b56b1a297"}
Oct 31 01:30:13.741289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.740803    5718 scope.go:122] "RemoveContainer" containerID="eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd"
Oct 31 01:30:13.753054 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.753034    5718 scope.go:122] "RemoveContainer" containerID="eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd"
Oct 31 01:30:13.753437 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:13.753412    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd\": container with ID starting with eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd not found: ID does not exist" containerID="eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd"
Oct 31 01:30:13.753565 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.753444    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd"} err="failed to get container status \"eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd\": rpc error: code = NotFound desc = could not find container \"eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd\": container with ID starting with eee6c10b5118978a85e4ec187ae88bd73042e60488d17ff1c3c8d0a892afbdbd not found: ID does not exist"
Oct 31 01:30:13.907932 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.907889    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"]
Oct 31 01:30:13.969354 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:13.969323    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18bf762c-5c7b6bc9d8-d2j5w"]
Oct 31 01:30:15.976620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:15.976580    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84517f1-e361-44ed-a104-88645e37ff3e" path="/var/lib/kubelet/pods/c84517f1-e361-44ed-a104-88645e37ff3e/volumes"
Oct 31 01:30:16.961226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:16.961192    5718 scope.go:122] "RemoveContainer" containerID="98f1648be43e6963d1e22f4b0475767f64d1fc852697c8bafe18407e6ead4ee4"
Oct 31 01:30:17.771604 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:17.771569    5718 generic.go:356] "Generic (PLEG): container finished" podID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerID="cf94f70ac3854bbdfb6b5e79d02f3721d26f2717615416a4fb60b8bebcfe6f14" exitCode=60
Oct 31 01:30:17.772034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:17.771608    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" event={"ID":"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb","Type":"ContainerDied","Data":"cf94f70ac3854bbdfb6b5e79d02f3721d26f2717615416a4fb60b8bebcfe6f14"}
Oct 31 01:30:17.772034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:17.771696    5718 scope.go:122] "RemoveContainer" containerID="98f1648be43e6963d1e22f4b0475767f64d1fc852697c8bafe18407e6ead4ee4"
Oct 31 01:30:17.772136 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:17.772118    5718 scope.go:122] "RemoveContainer" containerID="cf94f70ac3854bbdfb6b5e79d02f3721d26f2717615416a4fb60b8bebcfe6f14"
Oct 31 01:30:17.772494 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:17.772468    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-bp97r_odh-ai-gateway-infra(8cb47329-2b5d-4cc1-9643-2f51d2cbaddb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb"
Oct 31 01:30:17.961027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:17.960991    5718 scope.go:122] "RemoveContainer" containerID="3f69db07980f8b443afc24138959dbf475d851e94efad13e1d9f5b2fbb3f15ee"
Oct 31 01:30:18.783313 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:18.783269    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerStarted","Data":"eb397abe14b45823c1899702d4774ac340f8489699538124ffa95225ad0e1588"}
Oct 31 01:30:18.898057 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:18.898020    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"]
Oct 31 01:30:19.076364 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.076338    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:19.119326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.119288    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca\") pod \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") "
Oct 31 01:30:19.119564 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.119548    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p\" (UniqueName: \"kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p\") pod \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\" (UID: \"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb\") "
Oct 31 01:30:19.119749 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.119727    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca" pod "8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" (UID: "8cb47329-2b5d-4cc1-9643-2f51d2cbaddb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:19.119972 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.119959    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:19.122401 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.122371    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p" pod "8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" (UID: "8cb47329-2b5d-4cc1-9643-2f51d2cbaddb"). InnerVolumeSpecName "kube-api-access-fjz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:19.221584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.221540    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjz8p\" (UniqueName: \"kubernetes.io/projected/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb-kube-api-access-fjz8p\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:19.408834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.408737    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"]
Oct 31 01:30:19.410189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410161    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.410189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410188    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.410394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410254    5718 cpu_manager.go:414] "containerMap: removing container" podUID="c84517f1-e361-44ed-a104-88645e37ff3e" containerName="maas-api"
Oct 31 01:30:19.410394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410262    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84517f1-e361-44ed-a104-88645e37ff3e" containerName="maas-api"
Oct 31 01:30:19.410394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410273    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.410394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410280    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.410394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410305    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.410394 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410312    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.411129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410582    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.411129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410600    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.411129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410616    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84517f1-e361-44ed-a104-88645e37ff3e" containerName="maas-api"
Oct 31 01:30:19.411129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.410623    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" containerName="cleanup"
Oct 31 01:30:19.414082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.414056    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.423787 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.423760    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6nd\" (UniqueName: \"kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.423926 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.423823    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.423926 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.423921    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.424077 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.424056    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.469377 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.469343    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2c70e7-serving-cert\""
Oct 31 01:30:19.469803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.469774    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"]
Oct 31 01:30:19.525614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.525571    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.525869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.525746    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.525869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.525805    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.526000 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:19.525941    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-serving-cert: secret "maas-api-e2e-api-a-2c70e7-serving-cert" not found
Oct 31 01:30:19.526061 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:19.526021    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls podName:659ba115-de08-485a-9f4c-5af57725f7f0 nodeName:}" failed. No retries permitted until 2026-10-31 01:30:20.025999022 +0000 UTC m=+1536.189142419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls") pod "maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" (UID: "659ba115-de08-485a-9f4c-5af57725f7f0") : secret "maas-api-e2e-api-a-2c70e7-serving-cert" not found
Oct 31 01:30:19.526061 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.525951    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6nd\" (UniqueName: \"kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.526482 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.526461    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.528616 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.528590    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.584310 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.584277    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6nd\" (UniqueName: \"kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:19.793304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.793259    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r" event={"ID":"8cb47329-2b5d-4cc1-9643-2f51d2cbaddb","Type":"ContainerDied","Data":"dd11c5bacea7d3fa52cf9479f56205895fb8e96f89de0a944b9ff6fc4360be4c"}
Oct 31 01:30:19.793892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.793289    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"
Oct 31 01:30:19.793892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.793883    5718 scope.go:122] "RemoveContainer" containerID="cf94f70ac3854bbdfb6b5e79d02f3721d26f2717615416a4fb60b8bebcfe6f14"
Oct 31 01:30:19.796380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.796353    5718 generic.go:356] "Generic (PLEG): container finished" podID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerID="eb397abe14b45823c1899702d4774ac340f8489699538124ffa95225ad0e1588" exitCode=60
Oct 31 01:30:19.796508 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.796446    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerDied","Data":"eb397abe14b45823c1899702d4774ac340f8489699538124ffa95225ad0e1588"}
Oct 31 01:30:19.797009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.796989    5718 scope.go:122] "RemoveContainer" containerID="eb397abe14b45823c1899702d4774ac340f8489699538124ffa95225ad0e1588"
Oct 31 01:30:19.797374 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:19.797342    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45_odh-ai-gateway-infra(e884c4f9-2c44-4b57-8a17-59af1ac6f614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614"
Oct 31 01:30:19.812153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.812131    5718 scope.go:122] "RemoveContainer" containerID="3f69db07980f8b443afc24138959dbf475d851e94efad13e1d9f5b2fbb3f15ee"
Oct 31 01:30:19.911812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.911742    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"]
Oct 31 01:30:19.971304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.971272    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"]
Oct 31 01:30:19.974518 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:19.974493    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-bp97r"]
Oct 31 01:30:20.034112 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:20.034077    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:20.036697 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:20.036668    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\") pod \"maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:20.328604 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:20.328563    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:20.714507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:20.713992    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"]
Oct 31 01:30:20.806012 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:20.805977    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" event={"ID":"659ba115-de08-485a-9f4c-5af57725f7f0","Type":"ContainerStarted","Data":"573ce894713cfbc39ee3040c31d20702e45cf67200526de5a82072df70a5d302"}
Oct 31 01:30:20.963626 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:20.963569    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:21.051004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.050973    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s\" (UniqueName: \"kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s\") pod \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") "
Oct 31 01:30:21.051280 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.051262    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca\") pod \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\" (UID: \"e884c4f9-2c44-4b57-8a17-59af1ac6f614\") "
Oct 31 01:30:21.051586 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.051562    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca" pod "e884c4f9-2c44-4b57-8a17-59af1ac6f614" (UID: "e884c4f9-2c44-4b57-8a17-59af1ac6f614"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:21.051852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.051837    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884c4f9-2c44-4b57-8a17-59af1ac6f614-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:21.053401 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.053379    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s" pod "e884c4f9-2c44-4b57-8a17-59af1ac6f614" (UID: "e884c4f9-2c44-4b57-8a17-59af1ac6f614"). InnerVolumeSpecName "kube-api-access-6ds7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:21.153384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.153348    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ds7s\" (UniqueName: \"kubernetes.io/projected/e884c4f9-2c44-4b57-8a17-59af1ac6f614-kube-api-access-6ds7s\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:21.821050 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.821015    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45" event={"ID":"e884c4f9-2c44-4b57-8a17-59af1ac6f614","Type":"ContainerDied","Data":"9b05970a1769deb1cf8c3fa3a6b9b991db00d8c65ba89bc2dad19eaf456ca0ed"}
Oct 31 01:30:21.821050 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.821040    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"
Oct 31 01:30:21.821504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.821088    5718 scope.go:122] "RemoveContainer" containerID="eb397abe14b45823c1899702d4774ac340f8489699538124ffa95225ad0e1588"
Oct 31 01:30:21.968788 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.968758    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb47329-2b5d-4cc1-9643-2f51d2cbaddb" path="/var/lib/kubelet/pods/8cb47329-2b5d-4cc1-9643-2f51d2cbaddb/volumes"
Oct 31 01:30:21.991759 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.991721    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"]
Oct 31 01:30:21.993059 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:21.993039    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-42c96d-29890170-rdf45"]
Oct 31 01:30:22.828780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:22.828631    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" event={"ID":"659ba115-de08-485a-9f4c-5af57725f7f0","Type":"ContainerStarted","Data":"f663f74c732f07e2070b80ad8011ed1bfa353b059863586c248cb27b4d7e134a"}
Oct 31 01:30:22.829237 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:22.828916    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:22.947147 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:22.947089    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" podStartSLOduration=2.868680623 podStartE2EDuration="3.947072494s" totalImagesPullingTime="1.078391871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:22.945877013 +0000 UTC m=+1539.109020429" watchObservedRunningTime="2026-10-31 01:30:22.947072494 +0000 UTC m=+1539.110215941"
Oct 31 01:30:23.968426 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:23.968392    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" path="/var/lib/kubelet/pods/e884c4f9-2c44-4b57-8a17-59af1ac6f614/volumes"
Oct 31 01:30:24.203786 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:24.203748    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"]
Oct 31 01:30:24.203977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:24.203942    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy" containerID="cri-o://a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe" gracePeriod=30
Oct 31 01:30:24.255801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:24.255769    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"]
Oct 31 01:30:24.256034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:24.256008    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy" containerID="cri-o://8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba" gracePeriod=30
Oct 31 01:30:28.842839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:28.842801    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:30:29.601634 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.601593    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:30:29.611761 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.611738    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:30:29.678800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678773    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.678977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678815    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.678977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678837    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.678977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678867    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.678977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678894    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.678977 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678933    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.678984    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679002    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679016    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679055    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679087    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679154    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679182    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679201    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679215    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.679250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679222    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679262    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679325    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679369    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679418    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert\") pod \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\" (UID: \"2c06b2b8-2623-4b3d-9558-ed97794b2f9c\") "
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679426    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679509    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679511    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679610    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679685    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy\") pod \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\" (UID: \"4e5bb83a-d12b-4dc9-9365-c1050a60914b\") "
Oct 31 01:30:29.679817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679793    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.679946    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680328    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680350    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680366    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680380    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680395    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680409    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680495    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680595    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.680862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.680609    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.682222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.681912    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:29.682222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.681991    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "kube-api-access-bl9xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.682222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.682130    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.682453 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.682280    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:29.682453 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.682307    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "kube-api-access-krpnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.682453 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.682434    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.682994 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.682971    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.683071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.683046    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo" pod "4e5bb83a-d12b-4dc9-9365-c1050a60914b" (UID: "4e5bb83a-d12b-4dc9-9365-c1050a60914b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:29.683153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.683135    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy" pod "2c06b2b8-2623-4b3d-9558-ed97794b2f9c" (UID: "2c06b2b8-2623-4b3d-9558-ed97794b2f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782345    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782373    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782384    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782392    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782402    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782410    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl9xq\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-kube-api-access-bl9xq\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782420    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5bb83a-d12b-4dc9-9365-c1050a60914b-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782429    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krpnw\" (UniqueName: \"kubernetes.io/projected/4e5bb83a-d12b-4dc9-9365-c1050a60914b-kube-api-access-krpnw\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.782430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.782437    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c06b2b8-2623-4b3d-9558-ed97794b2f9c-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.880927 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.880888    5718 generic.go:356] "Generic (PLEG): container finished" podID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerID="8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba" exitCode=0
Oct 31 01:30:29.881405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.880941    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"
Oct 31 01:30:29.881405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.880983    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" event={"ID":"2c06b2b8-2623-4b3d-9558-ed97794b2f9c","Type":"ContainerDied","Data":"8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba"}
Oct 31 01:30:29.881405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.881056    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj" event={"ID":"2c06b2b8-2623-4b3d-9558-ed97794b2f9c","Type":"ContainerDied","Data":"1479df07e3516072a67a7a66afb1f22a880d3d194a8aef80ff747ac9c38f93d4"}
Oct 31 01:30:29.881405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.881089    5718 scope.go:122] "RemoveContainer" containerID="8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba"
Oct 31 01:30:29.883324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.883297    5718 generic.go:356] "Generic (PLEG): container finished" podID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerID="a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe" exitCode=0
Oct 31 01:30:29.883523 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.883504    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" event={"ID":"4e5bb83a-d12b-4dc9-9365-c1050a60914b","Type":"ContainerDied","Data":"a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe"}
Oct 31 01:30:29.883623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.883549    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp" event={"ID":"4e5bb83a-d12b-4dc9-9365-c1050a60914b","Type":"ContainerDied","Data":"32514c5d54683cfbbe97f64eb17c54c9f57c0a1a1050e2ab043241acd839876e"}
Oct 31 01:30:29.883735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.883629    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"
Oct 31 01:30:29.892426 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.892410    5718 scope.go:122] "RemoveContainer" containerID="8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba"
Oct 31 01:30:29.892770 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:29.892752    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba\": container with ID starting with 8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba not found: ID does not exist" containerID="8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba"
Oct 31 01:30:29.892834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.892778    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba"} err="failed to get container status \"8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba\": rpc error: code = NotFound desc = could not find container \"8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba\": container with ID starting with 8485b4ea716853d7b0e7d1e53927da9a09ee2c0f3e28cdea7bebcbdc1566c5ba not found: ID does not exist"
Oct 31 01:30:29.892834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.892795    5718 scope.go:122] "RemoveContainer" containerID="a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe"
Oct 31 01:30:29.902944 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.902928    5718 scope.go:122] "RemoveContainer" containerID="a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe"
Oct 31 01:30:29.903213 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:29.903191    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe\": container with ID starting with a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe not found: ID does not exist" containerID="a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe"
Oct 31 01:30:29.903305 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:29.903217    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe"} err="failed to get container status \"a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe\": rpc error: code = NotFound desc = could not find container \"a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe\": container with ID starting with a73b0337a4fca7a8eed21c54cdddb6845f213988e428dc7b1f9a21d4109264fe not found: ID does not exist"
Oct 31 01:30:30.047127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:30.047047    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"]
Oct 31 01:30:30.060169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:30.060130    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-whrlp"]
Oct 31 01:30:30.163851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:30.163806    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"]
Oct 31 01:30:30.167262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:30.167233    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-54bzj"]
Oct 31 01:30:31.967669 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:31.967629    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" path="/var/lib/kubelet/pods/2c06b2b8-2623-4b3d-9558-ed97794b2f9c/volumes"
Oct 31 01:30:31.968205 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:30:31.968185    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" path="/var/lib/kubelet/pods/4e5bb83a-d12b-4dc9-9365-c1050a60914b/volumes"
Oct 31 01:30:37.924877 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:30:37.924840    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:31:07.927496 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:31:07.927456    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:31:37.927581 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:31:37.927533    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:32:07.926349 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:32:07.926259    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:32:37.928748 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:32:37.928704    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:33:07.941059 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:33:07.941016    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:33:09.615494 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.615458    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"]
Oct 31 01:33:09.615942 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.615704    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" podUID="51fa7641-31ba-411c-b5ed-80f6e8996227" containerName="maas-api" containerID="cri-o://1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8" gracePeriod=30
Oct 31 01:33:09.974566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.974533    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:33:09.999421 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.999389    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca\") pod \"51fa7641-31ba-411c-b5ed-80f6e8996227\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") "
Oct 31 01:33:09.999593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.999464    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2\" (UniqueName: \"kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2\") pod \"51fa7641-31ba-411c-b5ed-80f6e8996227\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") "
Oct 31 01:33:09.999593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.999570    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs\") pod \"51fa7641-31ba-411c-b5ed-80f6e8996227\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") "
Oct 31 01:33:09.999816 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.999753    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\") pod \"51fa7641-31ba-411c-b5ed-80f6e8996227\" (UID: \"51fa7641-31ba-411c-b5ed-80f6e8996227\") "
Oct 31 01:33:09.999880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:09.999850    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca" pod "51fa7641-31ba-411c-b5ed-80f6e8996227" (UID: "51fa7641-31ba-411c-b5ed-80f6e8996227"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:10.000716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.000507    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51fa7641-31ba-411c-b5ed-80f6e8996227-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:10.002500 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.002454    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs" pod "51fa7641-31ba-411c-b5ed-80f6e8996227" (UID: "51fa7641-31ba-411c-b5ed-80f6e8996227"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:10.002753 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.002724    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2" pod "51fa7641-31ba-411c-b5ed-80f6e8996227" (UID: "51fa7641-31ba-411c-b5ed-80f6e8996227"). InnerVolumeSpecName "kube-api-access-24tp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:10.004964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.004935    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls" pod "51fa7641-31ba-411c-b5ed-80f6e8996227" (UID: "51fa7641-31ba-411c-b5ed-80f6e8996227"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:10.106067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.106032    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:10.106067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.106066    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24tp2\" (UniqueName: \"kubernetes.io/projected/51fa7641-31ba-411c-b5ed-80f6e8996227-kube-api-access-24tp2\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:10.106275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.106081    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51fa7641-31ba-411c-b5ed-80f6e8996227-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:10.117703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.116158    5718 generic.go:356] "Generic (PLEG): container finished" podID="51fa7641-31ba-411c-b5ed-80f6e8996227" containerID="1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8" exitCode=0
Oct 31 01:33:10.117703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.116365    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" event={"ID":"51fa7641-31ba-411c-b5ed-80f6e8996227","Type":"ContainerDied","Data":"1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8"}
Oct 31 01:33:10.117703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.116421    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn" event={"ID":"51fa7641-31ba-411c-b5ed-80f6e8996227","Type":"ContainerDied","Data":"a1fbf16911a146d8180d484fe4425d0f81ea3c4b6187aea3573ed160c447305e"}
Oct 31 01:33:10.117703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.116452    5718 scope.go:122] "RemoveContainer" containerID="1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8"
Oct 31 01:33:10.117703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.116637    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"
Oct 31 01:33:10.137748 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.137379    5718 scope.go:122] "RemoveContainer" containerID="1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8"
Oct 31 01:33:10.138445 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:33:10.138417    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8\": container with ID starting with 1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8 not found: ID does not exist" containerID="1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8"
Oct 31 01:33:10.138597 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.138458    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8"} err="failed to get container status \"1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8\": rpc error: code = NotFound desc = could not find container \"1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8\": container with ID starting with 1740603d8a16ca0a0fb5f2282c54c8d873ddc14580e80dc482e8f0879c3b59b8 not found: ID does not exist"
Oct 31 01:33:10.289138 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.289099    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"]
Oct 31 01:33:10.293569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:10.293535    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-01cbd8-6cc44b8d64-s95pn"]
Oct 31 01:33:11.972724 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:11.972695    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fa7641-31ba-411c-b5ed-80f6e8996227" path="/var/lib/kubelet/pods/51fa7641-31ba-411c-b5ed-80f6e8996227/volumes"
Oct 31 01:33:21.374181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:21.374149    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"]
Oct 31 01:33:21.374560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:21.374352    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy" containerID="cri-o://460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a" gracePeriod=30
Oct 31 01:33:26.654339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.654267    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.767948    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768025    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768070    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768193    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768239    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768318    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768395    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768497    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.768569    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket\") pod \"621b8137-c8de-42aa-8637-132c54e7e5cf\" (UID: \"621b8137-c8de-42aa-8637-132c54e7e5cf\") "
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.769219    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.769451    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.769590    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.769578    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.772609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.769781    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.801905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.801867    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.802068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.801896    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.802068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.802015    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "kube-api-access-lpqbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.808187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.808133    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo" pod "621b8137-c8de-42aa-8637-132c54e7e5cf" (UID: "621b8137-c8de-42aa-8637-132c54e7e5cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:26.870690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870636    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870684    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870695    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpqbh\" (UniqueName: \"kubernetes.io/projected/621b8137-c8de-42aa-8637-132c54e7e5cf-kube-api-access-lpqbh\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870709    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870718    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870728    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870737    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870746    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621b8137-c8de-42aa-8637-132c54e7e5cf-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.870936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:26.870754    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621b8137-c8de-42aa-8637-132c54e7e5cf-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:27.227612 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.227578    5718 generic.go:356] "Generic (PLEG): container finished" podID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerID="460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a" exitCode=0
Oct 31 01:33:27.227832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.227669    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"
Oct 31 01:33:27.227832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.227667    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" event={"ID":"621b8137-c8de-42aa-8637-132c54e7e5cf","Type":"ContainerDied","Data":"460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a"}
Oct 31 01:33:27.227832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.227789    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r" event={"ID":"621b8137-c8de-42aa-8637-132c54e7e5cf","Type":"ContainerDied","Data":"c886e7ee95c3b718e5a783ae9ba463ccf91633f19a65a1aaec01b3b16f36184b"}
Oct 31 01:33:27.227832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.227812    5718 scope.go:122] "RemoveContainer" containerID="460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a"
Oct 31 01:33:27.238202 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.238185    5718 scope.go:122] "RemoveContainer" containerID="460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a"
Oct 31 01:33:27.238470 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:33:27.238449    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a\": container with ID starting with 460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a not found: ID does not exist" containerID="460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a"
Oct 31 01:33:27.238554 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.238479    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a"} err="failed to get container status \"460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a\": rpc error: code = NotFound desc = could not find container \"460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a\": container with ID starting with 460fd432f8ea1cb6204f299fec95a1fdf85090dd973d45bece0c6ccd34b96f3a not found: ID does not exist"
Oct 31 01:33:27.396106 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.396069    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"]
Oct 31 01:33:27.446683 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.446621    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-wjw6r"]
Oct 31 01:33:27.968373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:27.968342    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" path="/var/lib/kubelet/pods/621b8137-c8de-42aa-8637-132c54e7e5cf/volumes"
Oct 31 01:33:28.976791 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:28.976760    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"]
Oct 31 01:33:28.977389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:28.977086    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy" containerID="cri-o://0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7" gracePeriod=30
Oct 31 01:33:34.256071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.256045    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:33:34.270584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.270554    5718 generic.go:356] "Generic (PLEG): container finished" podID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerID="0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7" exitCode=0
Oct 31 01:33:34.270777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.270697    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"
Oct 31 01:33:34.270777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.270697    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" event={"ID":"15f26f55-dc50-4e43-8c5c-da1a374a2c56","Type":"ContainerDied","Data":"0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7"}
Oct 31 01:33:34.270851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.270820    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw" event={"ID":"15f26f55-dc50-4e43-8c5c-da1a374a2c56","Type":"ContainerDied","Data":"f55911d959d72165a269ca60d50b1a5484e1ba5f22f4feed5d0113774db2cc25"}
Oct 31 01:33:34.270851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.270836    5718 scope.go:122] "RemoveContainer" containerID="0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7"
Oct 31 01:33:34.281621 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.281601    5718 scope.go:122] "RemoveContainer" containerID="0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7"
Oct 31 01:33:34.281982 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:33:34.281963    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7\": container with ID starting with 0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7 not found: ID does not exist" containerID="0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7"
Oct 31 01:33:34.282039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.281992    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7"} err="failed to get container status \"0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7\": rpc error: code = NotFound desc = could not find container \"0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7\": container with ID starting with 0431be301faf49fee6d3996cecd6d472db811ca37e6a4ced39d90e609ef7eca7 not found: ID does not exist"
Oct 31 01:33:34.331240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331208    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331297    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331317    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331362    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331423    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331439    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331518    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331519    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.331573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331565    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331642    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert\") pod \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\" (UID: \"15f26f55-dc50-4e43-8c5c-da1a374a2c56\") "
Oct 31 01:33:34.331869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331704    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.331869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331744    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.331869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.331849    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.332130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.332112    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:34.332488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.332466    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.332571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.332495    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.332571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.332510    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.332571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.332524    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.332571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.332539    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.333838 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.333814    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.333948 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.333887    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:34.334025 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.333972    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.334292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.334274    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll" pod "15f26f55-dc50-4e43-8c5c-da1a374a2c56" (UID: "15f26f55-dc50-4e43-8c5c-da1a374a2c56"). InnerVolumeSpecName "kube-api-access-dbmll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.434456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.434411    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.434456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.434445    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.434456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.434460    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26f55-dc50-4e43-8c5c-da1a374a2c56-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.434796 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.434473    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmll\" (UniqueName: \"kubernetes.io/projected/15f26f55-dc50-4e43-8c5c-da1a374a2c56-kube-api-access-dbmll\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.695853 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.695813    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"]
Oct 31 01:33:34.700024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:34.699999    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pj7nw"]
Oct 31 01:33:35.975538 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:33:35.975502    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" path="/var/lib/kubelet/pods/15f26f55-dc50-4e43-8c5c-da1a374a2c56/volumes"
Oct 31 01:33:37.927225 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:33:37.927185    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:34:07.925273 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:34:07.925230    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:34:36.955609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:36.955570    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"]
Oct 31 01:34:36.956146 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:36.955839    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy" containerID="cri-o://d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7" gracePeriod=30
Oct 31 01:34:37.926603 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:34:37.926566    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:34:38.743525 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:38.743494    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"]
Oct 31 01:34:38.744002 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:38.743706    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" containerID="cri-o://2f2769ddd99873aef644d2a34620e5a7b00c18eb4e8260a592ffbc929243f1e5" gracePeriod=30
Oct 31 01:34:42.224809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.224739    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:34:42.288090 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288055    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288261 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288132    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288282    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288391 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288334    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288440 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288410    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:42.288492 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288453    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288548 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288501    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:42.288548 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288522    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288567    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288586    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:42.288691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288637    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288841 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288708    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket\") pod \"838b972f-e0d5-42f7-b962-69c3cbd6de86\" (UID: \"838b972f-e0d5-42f7-b962-69c3cbd6de86\") "
Oct 31 01:34:42.288934 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.288909    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:42.289237 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.289164    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:42.289412 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.289392    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838b972f-e0d5-42f7-b962-69c3cbd6de86-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.289491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.289420    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.289491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.289433    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.289491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.289446    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.289491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.289460    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.290952 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.290925    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "kube-api-access-g7cgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:42.291055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.290973    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:42.291055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.291008    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:42.291169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.291119    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy" pod "838b972f-e0d5-42f7-b962-69c3cbd6de86" (UID: "838b972f-e0d5-42f7-b962-69c3cbd6de86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:42.391306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.391262    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.391306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.391296    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.391547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.391331    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7cgb\" (UniqueName: \"kubernetes.io/projected/838b972f-e0d5-42f7-b962-69c3cbd6de86-kube-api-access-g7cgb\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.391547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.391343    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838b972f-e0d5-42f7-b962-69c3cbd6de86-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:42.747008 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.746974    5718 generic.go:356] "Generic (PLEG): container finished" podID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerID="d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7" exitCode=0
Oct 31 01:34:42.747212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.747032    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"
Oct 31 01:34:42.747212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.747060    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" event={"ID":"838b972f-e0d5-42f7-b962-69c3cbd6de86","Type":"ContainerDied","Data":"d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7"}
Oct 31 01:34:42.747212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.747122    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm" event={"ID":"838b972f-e0d5-42f7-b962-69c3cbd6de86","Type":"ContainerDied","Data":"e1af94b622208938be3a6dd7117965ca650e09d071b1d4abcd0c0b88295305b9"}
Oct 31 01:34:42.747212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.747149    5718 scope.go:122] "RemoveContainer" containerID="d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7"
Oct 31 01:34:42.761497 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.761473    5718 scope.go:122] "RemoveContainer" containerID="d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7"
Oct 31 01:34:42.763345 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:34:42.763312    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7\": container with ID starting with d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7 not found: ID does not exist" containerID="d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7"
Oct 31 01:34:42.763472 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.763351    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7"} err="failed to get container status \"d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7\": rpc error: code = NotFound desc = could not find container \"d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7\": container with ID starting with d966011d4ac0ee16e761df2d94abdbb6c7d44016e559528c6b453147dc9691e7 not found: ID does not exist"
Oct 31 01:34:42.914169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.914134    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"]
Oct 31 01:34:42.917550 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:42.917520    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-n2drm"]
Oct 31 01:34:43.861451 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:43.861419    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 31 01:34:43.861844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:43.861487    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 31 01:34:43.970116 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:43.970081    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" path="/var/lib/kubelet/pods/838b972f-e0d5-42f7-b962-69c3cbd6de86/volumes"
Oct 31 01:34:44.022998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.022938    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:34:44.110256 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110225    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110464 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110313    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110464 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110369    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110464 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110406    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110511    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110576    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110620    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110628    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:44.110868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110674    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:44.110868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110692    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.110868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110728    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:44.110868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110766    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g\") pod \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\" (UID: \"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a\") "
Oct 31 01:34:44.111034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110964    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:44.111034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.110976    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:44.111567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.111540    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.111567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.111568    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.111931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.111584    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.111931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.111599    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.111931 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.111614    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.113063 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.113033    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:44.113176 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.113101    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:44.113220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.113190    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "kube-api-access-f5f4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:44.113260 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.113244    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token" pod "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" (UID: "3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:44.212714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.212681    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.212714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.212713    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.212967 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.212728    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5f4g\" (UniqueName: \"kubernetes.io/projected/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-kube-api-access-f5f4g\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.212967 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.212742    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:44.251747 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.251681    5718 scope.go:122] "RemoveContainer" containerID="2f2769ddd99873aef644d2a34620e5a7b00c18eb4e8260a592ffbc929243f1e5"
Oct 31 01:34:44.589123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.589028    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"]
Oct 31 01:34:44.589307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.589264    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" containerID="cri-o://700009f055edc0303ae8b07bcacccfbd9aa414cc48f99ae0f61c0864d0b9baac" gracePeriod=30
Oct 31 01:34:44.760778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.760738    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" event={"ID":"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a","Type":"ContainerDied","Data":"2f2769ddd99873aef644d2a34620e5a7b00c18eb4e8260a592ffbc929243f1e5"}
Oct 31 01:34:44.760966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.760779    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"
Oct 31 01:34:44.760966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.760797    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9" event={"ID":"3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a","Type":"ContainerDied","Data":"b88dab5f428a25dc78899c7839f1d030dbb7dbb191c4f91b0c2430b906e163f0"}
Oct 31 01:34:44.995110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.995012    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"]
Oct 31 01:34:44.997961 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:44.997932    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-bwbk9"]
Oct 31 01:34:45.968583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:45.968550    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" path="/var/lib/kubelet/pods/3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a/volumes"
Oct 31 01:34:46.894704 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:46.894629    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:34:46.895094 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:46.894728    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:34:49.795309 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.795268    5718 generic.go:356] "Generic (PLEG): container finished" podID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerID="700009f055edc0303ae8b07bcacccfbd9aa414cc48f99ae0f61c0864d0b9baac" exitCode=0
Oct 31 01:34:49.795710 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.795342    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" event={"ID":"93cdfe7f-f41c-4a1d-b947-75e324c0ea23","Type":"ContainerDied","Data":"700009f055edc0303ae8b07bcacccfbd9aa414cc48f99ae0f61c0864d0b9baac"}
Oct 31 01:34:49.850877 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.850849    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:34:49.987631 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.987530    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.987631 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.987612    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.987909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.987753    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.987909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.987787    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.987909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.987829    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.988111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.987940    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.988111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988007    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.988111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988042    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.988111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988090    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data\") pod \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\" (UID: \"93cdfe7f-f41c-4a1d-b947-75e324c0ea23\") "
Oct 31 01:34:49.988456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988402    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:49.988769 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988740    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:49.988873 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988828    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:49.988873 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.988845    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:49.989071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.989049    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:49.989387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.989360    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:49.990917 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.990890    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:49.991243 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.991220    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:49.991372 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.991347    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:49.991563 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:49.991544    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq" pod "93cdfe7f-f41c-4a1d-b947-75e324c0ea23" (UID: "93cdfe7f-f41c-4a1d-b947-75e324c0ea23"). InnerVolumeSpecName "kube-api-access-dd6sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:50.090514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090476    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090509    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090520    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090530    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090539    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6sq\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-kube-api-access-dd6sq\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090549    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090559    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.090805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.090569    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93cdfe7f-f41c-4a1d-b947-75e324c0ea23-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.806864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.806823    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx" event={"ID":"93cdfe7f-f41c-4a1d-b947-75e324c0ea23","Type":"ContainerDied","Data":"b3718517d03b77fe9a9e38d35d79e08e8e2e6fb9c023e875fe65c2eb49bf1727"}
Oct 31 01:34:50.807588 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.806888    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"
Oct 31 01:34:50.807588 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:50.806902    5718 scope.go:122] "RemoveContainer" containerID="700009f055edc0303ae8b07bcacccfbd9aa414cc48f99ae0f61c0864d0b9baac"
Oct 31 01:34:51.025215 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:51.025172    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"]
Oct 31 01:34:51.076229 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:51.076134    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tqjvx"]
Oct 31 01:34:51.967431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:51.967395    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" path="/var/lib/kubelet/pods/93cdfe7f-f41c-4a1d-b947-75e324c0ea23/volumes"
Oct 31 01:34:53.863182 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:53.863087    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"]
Oct 31 01:34:53.863624 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:53.863339    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" containerID="cri-o://4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056" gracePeriod=30
Oct 31 01:34:53.916306 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:53.916270    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"]
Oct 31 01:34:53.916532 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:53.916507    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" containerID="cri-o://a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34" gracePeriod=30
Oct 31 01:34:55.621557 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:55.621517    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"]
Oct 31 01:34:55.622030 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:55.621755    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" podUID="659ba115-de08-485a-9f4c-5af57725f7f0" containerName="maas-api" containerID="cri-o://f663f74c732f07e2070b80ad8011ed1bfa353b059863586c248cb27b4d7e134a" gracePeriod=30
Oct 31 01:34:55.873300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:55.873207    5718 generic.go:356] "Generic (PLEG): container finished" podID="659ba115-de08-485a-9f4c-5af57725f7f0" containerID="f663f74c732f07e2070b80ad8011ed1bfa353b059863586c248cb27b4d7e134a" exitCode=0
Oct 31 01:34:55.873588 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:55.873566    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" event={"ID":"659ba115-de08-485a-9f4c-5af57725f7f0","Type":"ContainerDied","Data":"f663f74c732f07e2070b80ad8011ed1bfa353b059863586c248cb27b4d7e134a"}
Oct 31 01:34:55.937964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:55.937936    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:34:56.083154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.083115    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs\") pod \"659ba115-de08-485a-9f4c-5af57725f7f0\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") "
Oct 31 01:34:56.083481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.083447    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca\") pod \"659ba115-de08-485a-9f4c-5af57725f7f0\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") "
Oct 31 01:34:56.083579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.083489    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\") pod \"659ba115-de08-485a-9f4c-5af57725f7f0\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") "
Oct 31 01:34:56.083637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.083582    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd\" (UniqueName: \"kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd\") pod \"659ba115-de08-485a-9f4c-5af57725f7f0\" (UID: \"659ba115-de08-485a-9f4c-5af57725f7f0\") "
Oct 31 01:34:56.084970 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.084942    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca" pod "659ba115-de08-485a-9f4c-5af57725f7f0" (UID: "659ba115-de08-485a-9f4c-5af57725f7f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:56.087024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.086993    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd" pod "659ba115-de08-485a-9f4c-5af57725f7f0" (UID: "659ba115-de08-485a-9f4c-5af57725f7f0"). InnerVolumeSpecName "kube-api-access-5m6nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:56.088055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.088033    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls" pod "659ba115-de08-485a-9f4c-5af57725f7f0" (UID: "659ba115-de08-485a-9f4c-5af57725f7f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:34:56.088142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.088109    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs" pod "659ba115-de08-485a-9f4c-5af57725f7f0" (UID: "659ba115-de08-485a-9f4c-5af57725f7f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:34:56.188189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.188027    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:56.188189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.188066    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ba115-de08-485a-9f4c-5af57725f7f0-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:56.188189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.188084    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ba115-de08-485a-9f4c-5af57725f7f0-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:56.188189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.188099    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m6nd\" (UniqueName: \"kubernetes.io/projected/659ba115-de08-485a-9f4c-5af57725f7f0-kube-api-access-5m6nd\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:56.881742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.881689    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b" event={"ID":"659ba115-de08-485a-9f4c-5af57725f7f0","Type":"ContainerDied","Data":"573ce894713cfbc39ee3040c31d20702e45cf67200526de5a82072df70a5d302"}
Oct 31 01:34:56.881742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.881729    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"
Oct 31 01:34:56.882326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:56.881780    5718 scope.go:122] "RemoveContainer" containerID="f663f74c732f07e2070b80ad8011ed1bfa353b059863586c248cb27b4d7e134a"
Oct 31 01:34:57.051520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:57.051477    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"]
Oct 31 01:34:57.100318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:57.100276    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c70e7-8689c6f686-cd99b"]
Oct 31 01:34:57.638951 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:57.638918    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:34:57.639126 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:57.638983    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:34:57.967124 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:57.967046    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659ba115-de08-485a-9f4c-5af57725f7f0" path="/var/lib/kubelet/pods/659ba115-de08-485a-9f4c-5af57725f7f0/volumes"
Oct 31 01:34:58.325754 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:58.325719    5718 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:34:58.326009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:58.325779    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:34:59.191302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.191278    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:34:59.209178 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.209155    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:34:59.227556 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227531    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.227727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227599    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.227727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227624    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.227727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227641    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.227727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227694    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227743    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227784    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227857    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227830    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227893    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227908    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227920    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.227958 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227950    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.227986    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228029    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228067    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228136    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket\") pod \"8856194b-a24b-4372-8d95-8dc546a9fa62\" (UID: \"8856194b-a24b-4372-8d95-8dc546a9fa62\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228142    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228175    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228235    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228256    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:59.228357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228264    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token\") pod \"753ea49d-2370-4c8f-bf9f-a32738d6369d\" (UID: \"753ea49d-2370-4c8f-bf9f-a32738d6369d\") "
Oct 31 01:34:59.229030 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.228417    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.230195 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230149    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.230400 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230373    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:59.230481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230460    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:59.230727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230705    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.230823 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230752    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:59.230866 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230834    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "kube-api-access-qwdh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:59.230954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230933    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.230954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.230940    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "kube-api-access-7w2xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:59.231278 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.231255    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.231427    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.231778    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8856194b-a24b-4372-8d95-8dc546a9fa62-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.231916    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.231932    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.231958    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232073    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2xb\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-kube-api-access-7w2xb\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232091    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232107    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232169    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/753ea49d-2370-4c8f-bf9f-a32738d6369d-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232187    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232203    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232224    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232338    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232352    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232369    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdh8\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-kube-api-access-qwdh8\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.232767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232532    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.233959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.232885    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy" pod "8856194b-a24b-4372-8d95-8dc546a9fa62" (UID: "8856194b-a24b-4372-8d95-8dc546a9fa62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.233959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.233481    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:59.233959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.233698    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token" pod "753ea49d-2370-4c8f-bf9f-a32738d6369d" (UID: "753ea49d-2370-4c8f-bf9f-a32738d6369d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:59.333941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.333903    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.333941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.333932    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/753ea49d-2370-4c8f-bf9f-a32738d6369d-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.333941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.333943    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8856194b-a24b-4372-8d95-8dc546a9fa62-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:59.902808 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.902768    5718 generic.go:356] "Generic (PLEG): container finished" podID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerID="4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056" exitCode=0
Oct 31 01:34:59.903009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.902846    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"
Oct 31 01:34:59.903009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.902853    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" event={"ID":"753ea49d-2370-4c8f-bf9f-a32738d6369d","Type":"ContainerDied","Data":"4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056"}
Oct 31 01:34:59.903009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.902949    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj" event={"ID":"753ea49d-2370-4c8f-bf9f-a32738d6369d","Type":"ContainerDied","Data":"20924de8e37fe1a08dcc01708822889d3981f603add6f7ce0278b6183c66ce61"}
Oct 31 01:34:59.903187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.902981    5718 scope.go:122] "RemoveContainer" containerID="4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056"
Oct 31 01:34:59.904591 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.904566    5718 generic.go:356] "Generic (PLEG): container finished" podID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerID="a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34" exitCode=0
Oct 31 01:34:59.904842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.904822    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" event={"ID":"8856194b-a24b-4372-8d95-8dc546a9fa62","Type":"ContainerDied","Data":"a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34"}
Oct 31 01:34:59.904902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.904872    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8" event={"ID":"8856194b-a24b-4372-8d95-8dc546a9fa62","Type":"ContainerDied","Data":"0c7dd448790a47328281f92ed842768cab16b9cd1847fa708d284ed3fe70f1cb"}
Oct 31 01:34:59.904969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.904957    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"
Oct 31 01:34:59.914506 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.914480    5718 scope.go:122] "RemoveContainer" containerID="4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056"
Oct 31 01:34:59.914832 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:34:59.914803    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056\": container with ID starting with 4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056 not found: ID does not exist" containerID="4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056"
Oct 31 01:34:59.914904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.914844    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056"} err="failed to get container status \"4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056\": rpc error: code = NotFound desc = could not find container \"4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056\": container with ID starting with 4ac290212ff5ea06410c98eae2af967ca8f7915c057e49431a4081c2c30dc056 not found: ID does not exist"
Oct 31 01:34:59.914904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.914866    5718 scope.go:122] "RemoveContainer" containerID="a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34"
Oct 31 01:34:59.924020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.924001    5718 scope.go:122] "RemoveContainer" containerID="a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34"
Oct 31 01:34:59.924313 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:34:59.924293    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34\": container with ID starting with a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34 not found: ID does not exist" containerID="a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34"
Oct 31 01:34:59.924374 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:34:59.924322    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34"} err="failed to get container status \"a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34\": rpc error: code = NotFound desc = could not find container \"a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34\": container with ID starting with a0005e0fa80a2e383492f401a36436e389382f4e536564a6403f82a47e664d34 not found: ID does not exist"
Oct 31 01:35:00.069905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:35:00.069869    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"]
Oct 31 01:35:00.119963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:35:00.119929    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-lqvz8"]
Oct 31 01:35:00.184904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:35:00.184817    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"]
Oct 31 01:35:00.190044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:35:00.190017    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-67wvj"]
Oct 31 01:35:01.967495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:35:01.967462    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" path="/var/lib/kubelet/pods/753ea49d-2370-4c8f-bf9f-a32738d6369d/volumes"
Oct 31 01:35:01.967980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:35:01.967965    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" path="/var/lib/kubelet/pods/8856194b-a24b-4372-8d95-8dc546a9fa62/volumes"
Oct 31 01:35:07.923644 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:35:07.923604    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:35:37.929519 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:35:37.929444    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:36:07.924682 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:36:07.924622    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:36:30.404750 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:30.404712    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"]
Oct 31 01:36:30.405328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:30.404944    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" containerID="cri-o://cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab" gracePeriod=30
Oct 31 01:36:34.074874 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:34.074831    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:36:34.075346 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:34.074897    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:36:35.668672 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.668628    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:36:35.790130 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790096    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790183    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790218    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790305    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790322    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790553 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790345    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790553 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790387    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790553 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790399    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.790553 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790435    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790553 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790540    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token\") pod \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\" (UID: \"a6fdc068-5f83-45e0-8655-3bfb0040e0a4\") "
Oct 31 01:36:35.790912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790626    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.790912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790644    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.790912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790720    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:35.790912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.790845    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.791163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.791143    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.791232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.791166    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.791232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.791176    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.791232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.791187    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.791232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.791197    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.792910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.792869    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:35.792910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.792886    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:35.793074 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.793025    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.793111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.793078    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf" pod "a6fdc068-5f83-45e0-8655-3bfb0040e0a4" (UID: "a6fdc068-5f83-45e0-8655-3bfb0040e0a4"). InnerVolumeSpecName "kube-api-access-lwnlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:35.891963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.891926    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.891963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.891959    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwnlf\" (UniqueName: \"kubernetes.io/projected/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-kube-api-access-lwnlf\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.891963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.891969    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.892202 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:35.891979    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fdc068-5f83-45e0-8655-3bfb0040e0a4-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:36.511096 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.511044    5718 generic.go:356] "Generic (PLEG): container finished" podID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerID="cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab" exitCode=0
Oct 31 01:36:36.511295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.511109    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" event={"ID":"a6fdc068-5f83-45e0-8655-3bfb0040e0a4","Type":"ContainerDied","Data":"cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab"}
Oct 31 01:36:36.511295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.511130    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"
Oct 31 01:36:36.511295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.511189    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6" event={"ID":"a6fdc068-5f83-45e0-8655-3bfb0040e0a4","Type":"ContainerDied","Data":"f5fd88cb5df18f21ac974f7de574d0526da80fe3e1755332c3e4c1352abdb404"}
Oct 31 01:36:36.511295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.511222    5718 scope.go:122] "RemoveContainer" containerID="cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab"
Oct 31 01:36:36.530107 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.530079    5718 scope.go:122] "RemoveContainer" containerID="cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab"
Oct 31 01:36:36.530833 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:36:36.530802    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab\": container with ID starting with cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab not found: ID does not exist" containerID="cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab"
Oct 31 01:36:36.530950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.530846    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab"} err="failed to get container status \"cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab\": rpc error: code = NotFound desc = could not find container \"cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab\": container with ID starting with cf67d77732417ab3c666d3c51d9b219586761af76e308bf247eac5b1cc9b66ab not found: ID does not exist"
Oct 31 01:36:36.729389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.729350    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"]
Oct 31 01:36:36.731996 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:36.731970    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-79cg6"]
Oct 31 01:36:37.924423 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:36:37.924383    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:36:37.967950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:36:37.967919    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" path="/var/lib/kubelet/pods/a6fdc068-5f83-45e0-8655-3bfb0040e0a4/volumes"
Oct 31 01:37:00.656474 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:00.656435    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"]
Oct 31 01:37:00.657078 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:00.656726    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" containerID="cri-o://03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c" gracePeriod=30
Oct 31 01:37:04.397247 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:04.397211    5718 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:37:04.397609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:04.397279    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:37:05.993822 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:05.993788    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:37:06.104104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104071    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104136    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104185    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104219    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104244    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104320 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104301    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104492    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104563    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104627    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs\") pod \"3e543206-2029-4141-b817-28b3fc200502\" (UID: \"3e543206-2029-4141-b817-28b3fc200502\") "
Oct 31 01:37:06.104752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104607    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:06.104752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104731    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:37:06.104962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.104941    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:06.105201 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.105180    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.105301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.105206    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e543206-2029-4141-b817-28b3fc200502-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.105301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.105222    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.105301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.105097    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:06.105301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.105200    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:06.106702 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.106674    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:37:06.107148 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.107124    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:06.107272 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.107252    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "kube-api-access-j569g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:06.107334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.107250    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy" pod "3e543206-2029-4141-b817-28b3fc200502" (UID: "3e543206-2029-4141-b817-28b3fc200502"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:37:06.206411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.206318    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.206411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.206355    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.206411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.206370    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e543206-2029-4141-b817-28b3fc200502-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.206411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.206383    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.206411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.206395    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e543206-2029-4141-b817-28b3fc200502-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.206411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.206409    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j569g\" (UniqueName: \"kubernetes.io/projected/3e543206-2029-4141-b817-28b3fc200502-kube-api-access-j569g\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:06.717303 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.717266    5718 generic.go:356] "Generic (PLEG): container finished" podID="3e543206-2029-4141-b817-28b3fc200502" containerID="03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c" exitCode=0
Oct 31 01:37:06.717497 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.717316    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"
Oct 31 01:37:06.717497 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.717347    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" event={"ID":"3e543206-2029-4141-b817-28b3fc200502","Type":"ContainerDied","Data":"03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c"}
Oct 31 01:37:06.717497 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.717400    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l" event={"ID":"3e543206-2029-4141-b817-28b3fc200502","Type":"ContainerDied","Data":"1b47ed528ed827a9a656e772d9b1038c758af3b78660b227bc3d8d1fbddc0887"}
Oct 31 01:37:06.717497 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.717423    5718 scope.go:122] "RemoveContainer" containerID="03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c"
Oct 31 01:37:06.729377 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.729356    5718 scope.go:122] "RemoveContainer" containerID="03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c"
Oct 31 01:37:06.729668 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:37:06.729632    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c\": container with ID starting with 03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c not found: ID does not exist" containerID="03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c"
Oct 31 01:37:06.729737 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.729682    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c"} err="failed to get container status \"03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c\": rpc error: code = NotFound desc = could not find container \"03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c\": container with ID starting with 03cbdf6af35f810722264253276aba590451c582bb83acf0ea088c6f0856837c not found: ID does not exist"
Oct 31 01:37:06.935927 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.935896    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"]
Oct 31 01:37:06.986244 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:06.986151    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-2f52l"]
Oct 31 01:37:07.923791 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:37:07.923753    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:37:07.966918 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:37:07.966884    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e543206-2029-4141-b817-28b3fc200502" path="/var/lib/kubelet/pods/3e543206-2029-4141-b817-28b3fc200502/volumes"
Oct 31 01:37:37.926374 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:37:37.926334    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:38:07.924348 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:38:07.924260    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:38:08.622248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:08.622205    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"]
Oct 31 01:38:08.622509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:08.622483    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" containerID="cri-o://5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84" gracePeriod=30
Oct 31 01:38:12.115547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:12.115515    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:38:12.115964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:12.115561    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:38:13.900067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:13.899994    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:38:14.039762 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039722    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.039940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039781    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.039940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039848    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.039940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039870    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.039940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039908    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.039940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039931    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.040223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.039998    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.040223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040091    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.040223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040152    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo\") pod \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\" (UID: \"a9bfe2b5-f59c-4f4b-9295-eadfcc164548\") "
Oct 31 01:38:14.040223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040174    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:14.040430 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040408    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:14.040486 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040422    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:14.040486 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040425    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:14.040685 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040626    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:14.040847 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040748    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.040847 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040771    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.040847 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040785    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.040847 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.040798    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.042480 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.042449    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:38:14.042587 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.042531    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:14.042762 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.042738    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:14.042848 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.042763    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn" pod "a9bfe2b5-f59c-4f4b-9295-eadfcc164548" (UID: "a9bfe2b5-f59c-4f4b-9295-eadfcc164548"). InnerVolumeSpecName "kube-api-access-95hmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:14.130665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.130618    5718 generic.go:356] "Generic (PLEG): container finished" podID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerID="5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84" exitCode=0
Oct 31 01:38:14.130842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.130699    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"
Oct 31 01:38:14.130842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.130710    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" event={"ID":"a9bfe2b5-f59c-4f4b-9295-eadfcc164548","Type":"ContainerDied","Data":"5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84"}
Oct 31 01:38:14.130842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.130769    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg" event={"ID":"a9bfe2b5-f59c-4f4b-9295-eadfcc164548","Type":"ContainerDied","Data":"388cd43e5608feffb7f22fa330f5c14b1b98185923e5ef584b4333567eff741c"}
Oct 31 01:38:14.130842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.130798    5718 scope.go:122] "RemoveContainer" containerID="5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84"
Oct 31 01:38:14.140887 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.140864    5718 scope.go:122] "RemoveContainer" containerID="5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84"
Oct 31 01:38:14.141209 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:38:14.141186    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84\": container with ID starting with 5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84 not found: ID does not exist" containerID="5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84"
Oct 31 01:38:14.141284 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.141222    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84"} err="failed to get container status \"5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84\": rpc error: code = NotFound desc = could not find container \"5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84\": container with ID starting with 5681840f38a9304f8e1231805d887af4109e4c27309b12d6f14d9f4d92f4db84 not found: ID does not exist"
Oct 31 01:38:14.141569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.141554    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.141606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.141573    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hmn\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-kube-api-access-95hmn\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.141606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.141594    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.141606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.141603    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.141787 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.141612    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bfe2b5-f59c-4f4b-9295-eadfcc164548-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:14.296225 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.296189    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"]
Oct 31 01:38:14.300579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:14.300548    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-jb2zg"]
Oct 31 01:38:15.968106 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:15.968073    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" path="/var/lib/kubelet/pods/a9bfe2b5-f59c-4f4b-9295-eadfcc164548/volumes"
Oct 31 01:38:32.553384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553351    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"]
Oct 31 01:38:32.553885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553870    5718 cpu_manager.go:414] "containerMap: removing container" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy"
Oct 31 01:38:32.553885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553886    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553897    5718 cpu_manager.go:414] "containerMap: removing container" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553902    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553911    5718 cpu_manager.go:414] "containerMap: removing container" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553917    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553926    5718 cpu_manager.go:414] "containerMap: removing container" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553931    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553939    5718 cpu_manager.go:414] "containerMap: removing container" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553943    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553949    5718 cpu_manager.go:414] "containerMap: removing container" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553953    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553960    5718 cpu_manager.go:414] "containerMap: removing container" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy"
Oct 31 01:38:32.553962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553965    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553974    5718 cpu_manager.go:414] "containerMap: removing container" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553979    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.553996    5718 cpu_manager.go:414] "containerMap: removing container" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554001    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554009    5718 cpu_manager.go:414] "containerMap: removing container" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554016    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554026    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554033    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554051    5718 cpu_manager.go:414] "containerMap: removing container" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554055    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554072    5718 cpu_manager.go:414] "containerMap: removing container" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554078    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554093    5718 cpu_manager.go:414] "containerMap: removing container" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554100    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554112    5718 cpu_manager.go:414] "containerMap: removing container" podUID="51fa7641-31ba-411c-b5ed-80f6e8996227" containerName="maas-api"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554118    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fa7641-31ba-411c-b5ed-80f6e8996227" containerName="maas-api"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554134    5718 cpu_manager.go:414] "containerMap: removing container" podUID="659ba115-de08-485a-9f4c-5af57725f7f0" containerName="maas-api"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554142    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659ba115-de08-485a-9f4c-5af57725f7f0" containerName="maas-api"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554262    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5bb83a-d12b-4dc9-9365-c1050a60914b" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554275    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e543206-2029-4141-b817-28b3fc200502" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554284    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="838b972f-e0d5-42f7-b962-69c3cbd6de86" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554291    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554299    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d66d6fd-8212-4635-bbc4-2d6a4d0c9a5a" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554310    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f26f55-dc50-4e43-8c5c-da1a374a2c56" containerName="istio-proxy"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554318    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fa7641-31ba-411c-b5ed-80f6e8996227" containerName="maas-api"
Oct 31 01:38:32.554318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554328    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554338    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fdc068-5f83-45e0-8655-3bfb0040e0a4" containerName="istio-proxy"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554346    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8856194b-a24b-4372-8d95-8dc546a9fa62" containerName="istio-proxy"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554355    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="753ea49d-2370-4c8f-bf9f-a32738d6369d" containerName="istio-proxy"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554363    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="659ba115-de08-485a-9f4c-5af57725f7f0" containerName="maas-api"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554373    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c06b2b8-2623-4b3d-9558-ed97794b2f9c" containerName="istio-proxy"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554381    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cdfe7f-f41c-4a1d-b947-75e324c0ea23" containerName="istio-proxy"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554388    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bfe2b5-f59c-4f4b-9295-eadfcc164548" containerName="istio-proxy"
Oct 31 01:38:32.555073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.554395    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="621b8137-c8de-42aa-8637-132c54e7e5cf" containerName="istio-proxy"
Oct 31 01:38:32.559389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.559364    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.614476 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.614447    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-447154-serving-cert\""
Oct 31 01:38:32.663686 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.663634    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwb84\" (UniqueName: \"kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.663874 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.663786    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.663941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.663923    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.663986 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.663980    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.667818 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.667795    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"]
Oct 31 01:38:32.764673 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.764620    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwb84\" (UniqueName: \"kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.764858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.764727    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.764858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.764846    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.764982 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:38:32.764872    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-serving-cert: secret "maas-api-e2e-ipp-b-447154-serving-cert" not found
Oct 31 01:38:32.764982 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:38:32.764936    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls podName:9fed430c-ca33-4e1a-b778-730f97456126 nodeName:}" failed. No retries permitted until 2026-10-31 01:38:33.264915666 +0000 UTC m=+2029.428059077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls") pod "maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" (UID: "9fed430c-ca33-4e1a-b778-730f97456126") : secret "maas-api-e2e-ipp-b-447154-serving-cert" not found
Oct 31 01:38:32.764982 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.764953    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.765571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.765540    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.767420 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.767396    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:32.825559 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:32.825491    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwb84\" (UniqueName: \"kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:33.270410 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:33.270372    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:33.273069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:33.273043    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\") pod \"maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:33.472288 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:33.472253    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:33.846999 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:33.846962    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"]
Oct 31 01:38:33.848461 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:38:33.848433    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fed430c_ca33_4e1a_b778_730f97456126.slice/crio-9d6569e9eede3fcca4e76fb9669c2000ddde164a90e8437e2262182d68578511 WatchSource:0}: Error finding container 9d6569e9eede3fcca4e76fb9669c2000ddde164a90e8437e2262182d68578511: Status 404 returned error can't find the container with id 9d6569e9eede3fcca4e76fb9669c2000ddde164a90e8437e2262182d68578511
Oct 31 01:38:33.850254 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:33.850232    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:38:34.256772 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:34.256741    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" event={"ID":"9fed430c-ca33-4e1a-b778-730f97456126","Type":"ContainerStarted","Data":"9d6569e9eede3fcca4e76fb9669c2000ddde164a90e8437e2262182d68578511"}
Oct 31 01:38:35.266002 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:35.265966    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" event={"ID":"9fed430c-ca33-4e1a-b778-730f97456126","Type":"ContainerStarted","Data":"9d0564f298141a1f179e040c3427ae3160f8e750184cc6be7f66deb9781e4bd0"}
Oct 31 01:38:35.266428 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:35.266077    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:38:35.382726 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:35.382620    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" podStartSLOduration=2.488207702 podStartE2EDuration="3.382599598s" totalImagesPullingTime="894.391896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:35.382092949 +0000 UTC m=+2031.545236364" watchObservedRunningTime="2026-10-31 01:38:35.382599598 +0000 UTC m=+2031.545743016"
Oct 31 01:38:37.924085 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:38:37.924047    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:38:41.277173 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:38:41.277143    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:39:07.925909 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:39:07.925868    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:39:37.924049 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:39:37.924006    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:40:07.924617 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:40:07.924573    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:40:37.924129 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:40:37.924089    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:40:50.323027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.322982    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"]
Oct 31 01:40:50.323583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.323500    5718 cpu_manager.go:414] "containerMap: removing container" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:40:50.323583 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.323513    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:40:50.323691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.323610    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="e884c4f9-2c44-4b57-8a17-59af1ac6f614" containerName="cleanup"
Oct 31 01:40:50.326940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.326924    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.379753 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.379719    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8af859-openshift-default-dockercfg-2ft7r\""
Oct 31 01:40:50.434980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.434943    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc8w\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435173 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.434999    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435173 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435084    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435188    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435229    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435255    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435426 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435390    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435423    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.435527 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.435510    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.536048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.536015    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"]
Oct 31 01:40:50.536727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.536694    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc8w\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.536868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.536788    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.536868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.536819    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.536987 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.536913    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.536987 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.536963    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537001    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537049    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537083    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537264 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537114    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537351    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537522 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537391    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537677 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537641    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537683    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.537810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.537766    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.540294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.540250    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.542154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.542125    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.594844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.594761    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.594990 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.594854    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc8w\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:50.640517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:50.640477    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:51.072750 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:40:51.072718    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7897809_3e59_4b95_97d5_619a3e00e9f4.slice/crio-9a39d2de6201086bf47a2d20de31543a946deb0320f954dfd69e0f51c466ef85 WatchSource:0}: Error finding container 9a39d2de6201086bf47a2d20de31543a946deb0320f954dfd69e0f51c466ef85: Status 404 returned error can't find the container with id 9a39d2de6201086bf47a2d20de31543a946deb0320f954dfd69e0f51c466ef85
Oct 31 01:40:51.075853 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:51.075804    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:40:51.075969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:51.075897    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:40:51.075969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:51.075944    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:40:51.076450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:51.076393    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"]
Oct 31 01:40:52.087515 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:52.087470    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" event={"ID":"f7897809-3e59-4b95-97d5-619a3e00e9f4","Type":"ContainerStarted","Data":"60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1"}
Oct 31 01:40:52.087927 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:52.087532    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" event={"ID":"f7897809-3e59-4b95-97d5-619a3e00e9f4","Type":"ContainerStarted","Data":"9a39d2de6201086bf47a2d20de31543a946deb0320f954dfd69e0f51c466ef85"}
Oct 31 01:40:52.259953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:52.259882    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podStartSLOduration=2.259859352 podStartE2EDuration="2.259859352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:52.259279026 +0000 UTC m=+2168.422422450" watchObservedRunningTime="2026-10-31 01:40:52.259859352 +0000 UTC m=+2168.423002772"
Oct 31 01:40:52.661741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:52.640707    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:52.662799 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:52.662764    5718 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:40:52.663030 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:52.663000    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 01:40:53.663678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:53.645076    5718 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:40:53.663678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:53.645168    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 01:40:54.641024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:54.640986    5718 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:40:54.641344 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:54.641312    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 01:40:55.640885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:55.640847    5718 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:40:55.641672 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:55.641615    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 01:40:56.657073 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:56.657035    5718 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:40:56.657716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:56.657680    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 01:40:57.646297 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:57.646261    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:57.646764 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:57.646738    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:40:57.647507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:40:57.647486    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:41:06.762129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.762085    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"]
Oct 31 01:41:06.769044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.769017    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:06.847027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.846992    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"]
Oct 31 01:41:06.869601 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.869546    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-447154-dockercfg-24hrf\""
Oct 31 01:41:06.869601 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.869562    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-447154\""
Oct 31 01:41:06.870088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.870065    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5svkk\" (UniqueName: \"kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk\") pod \"payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:06.870228 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.870204    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:06.971787 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.971749    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5svkk\" (UniqueName: \"kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk\") pod \"payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:06.972000 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.971843    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:06.972613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:06.972584    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:07.031000 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.030920    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5svkk\" (UniqueName: \"kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk\") pod \"payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:07.151574 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.151533    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:07.527706 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.527676    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"]
Oct 31 01:41:07.529056 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:41:07.529024    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901612dd_b099_4d9b_b410_f9c465dd5798.slice/crio-c26a85eeae92296cf13d0981157baf3bb3e2412c460bc367f4238b15c3194b7e WatchSource:0}: Error finding container c26a85eeae92296cf13d0981157baf3bb3e2412c460bc367f4238b15c3194b7e: Status 404 returned error can't find the container with id c26a85eeae92296cf13d0981157baf3bb3e2412c460bc367f4238b15c3194b7e
Oct 31 01:41:07.556186 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.556146    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"]
Oct 31 01:41:07.561226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.561206    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.580350 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.580320    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8d52\" (UniqueName: \"kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52\") pod \"payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.580577 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.580512    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.618422 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.618392    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"]
Oct 31 01:41:07.682422 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.682386    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8d52\" (UniqueName: \"kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52\") pod \"payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.683207 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.683180    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.684455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.684184    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.740438 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.740405    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8d52\" (UniqueName: \"kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52\") pod \"payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.875758 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:07.875646    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:07.926530 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:41:07.926490    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:41:08.203199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:08.203112    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl" event={"ID":"901612dd-b099-4d9b-b410-f9c465dd5798","Type":"ContainerStarted","Data":"c26a85eeae92296cf13d0981157baf3bb3e2412c460bc367f4238b15c3194b7e"}
Oct 31 01:41:08.251641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:08.251607    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"]
Oct 31 01:41:08.253514 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:41:08.253484    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507dfb77_eaae_4de5_991b_2bf8607e3475.slice/crio-7a869d6426f02cddf3da19af2e4c238362a7b6d5f8843b4d78ef70ce20387dd8 WatchSource:0}: Error finding container 7a869d6426f02cddf3da19af2e4c238362a7b6d5f8843b4d78ef70ce20387dd8: Status 404 returned error can't find the container with id 7a869d6426f02cddf3da19af2e4c238362a7b6d5f8843b4d78ef70ce20387dd8
Oct 31 01:41:09.209989 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:09.209946    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx" event={"ID":"507dfb77-eaae-4de5-991b-2bf8607e3475","Type":"ContainerStarted","Data":"7a869d6426f02cddf3da19af2e4c238362a7b6d5f8843b4d78ef70ce20387dd8"}
Oct 31 01:41:10.218885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:10.218843    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl" event={"ID":"901612dd-b099-4d9b-b410-f9c465dd5798","Type":"ContainerStarted","Data":"e6d8755462f37cdf9f85a2246216752d75a2566d65fd74e8dc09388ac826f920"}
Oct 31 01:41:10.219385 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:10.219030    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:10.220830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:10.220802    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx" event={"ID":"507dfb77-eaae-4de5-991b-2bf8607e3475","Type":"ContainerStarted","Data":"34b36d736c1958ccb9010e2c6e9cfbc85be5df468c8b4209a2021f438df18398"}
Oct 31 01:41:10.220963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:10.220933    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:10.341308 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:10.341194    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl" podStartSLOduration=2.098634218 podStartE2EDuration="4.341176818s" totalImagesPullingTime="2.2425426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:10.339017836 +0000 UTC m=+2186.502161254" watchObservedRunningTime="2026-10-31 01:41:10.341176818 +0000 UTC m=+2186.504320251"
Oct 31 01:41:10.462724 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:10.458931    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx" podStartSLOduration=1.7464183530000001 podStartE2EDuration="3.458909639s" totalImagesPullingTime="1.712491286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:10.456482679 +0000 UTC m=+2186.619626094" watchObservedRunningTime="2026-10-31 01:41:10.458909639 +0000 UTC m=+2186.622053056"
Oct 31 01:41:17.618197 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:17.618159    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"]
Oct 31 01:41:17.618814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:17.618394    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy" containerID="cri-o://be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8" gracePeriod=30
Oct 31 01:41:21.228816 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:21.228786    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:41:21.229314 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:21.228857    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:41:22.911834 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:22.911806    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:41:23.077620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.077587    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.077860 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.077777    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.077860 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.077819    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.077860 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.077847    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.078031 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.077897    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.078031 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.077991    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.078131 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078037    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.078131 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078104    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.078231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078137    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo\") pod \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\" (UID: \"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2\") "
Oct 31 01:41:23.078231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078147    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:23.078231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078134    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:23.078231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078204    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:23.079164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078586    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:23.079164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078734    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:23.079164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078740    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.079164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078781    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.079164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078800    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.079164 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.078817    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.080350 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.080326    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:23.080476 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.080339    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:23.080666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.080618    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "kube-api-access-qpxmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:23.082529 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.082507    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo" pod "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" (UID: "f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:41:23.180102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.180018    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.180102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.180049    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxmf\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-kube-api-access-qpxmf\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.180102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.180060    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.180102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.180070    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.180102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.180079    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:23.297222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.297182    5718 generic.go:356] "Generic (PLEG): container finished" podID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerID="be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8" exitCode=0
Oct 31 01:41:23.297415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.297237    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"
Oct 31 01:41:23.297415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.297266    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" event={"ID":"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2","Type":"ContainerDied","Data":"be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8"}
Oct 31 01:41:23.297415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.297323    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj" event={"ID":"f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2","Type":"ContainerDied","Data":"3b7efb5504c1fbe00b67766abaaea1e4ccff6bfd5e3ca9c0e657d324c35da577"}
Oct 31 01:41:23.297415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.297344    5718 scope.go:122] "RemoveContainer" containerID="be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8"
Oct 31 01:41:23.307984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.307968    5718 scope.go:122] "RemoveContainer" containerID="be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8"
Oct 31 01:41:23.308251 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:41:23.308233    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8\": container with ID starting with be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8 not found: ID does not exist" containerID="be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8"
Oct 31 01:41:23.308302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.308261    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8"} err="failed to get container status \"be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8\": rpc error: code = NotFound desc = could not find container \"be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8\": container with ID starting with be97a2bd45389cf3940ffa688abb5d86ac757708eeb771c17ebda79a27edc7c8 not found: ID does not exist"
Oct 31 01:41:23.463055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.462972    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"]
Oct 31 01:41:23.466206 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.466176    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rzxj"]
Oct 31 01:41:23.967165 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:23.967133    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" path="/var/lib/kubelet/pods/f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2/volumes"
Oct 31 01:41:37.924065 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:41:37.924019    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:41:53.762111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:53.762074    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"]
Oct 31 01:41:53.762582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:53.762273    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy" containerID="cri-o://4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950" gracePeriod=30
Oct 31 01:41:59.036447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.036383    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:41:59.121973 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.121935    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.121973 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.121978    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122000    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122085    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122146    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122190 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122165    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122205    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122264    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:59.122407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122274    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122357    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket\") pod \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\" (UID: \"2d5314ae-9bd7-44c1-9345-73dddd0b6723\") "
Oct 31 01:41:59.122628 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122475    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:59.122628 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122599    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:59.122766 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122635    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:59.122890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.122837    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:59.123184 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.123162    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.123301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.123188    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.123301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.123204    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.123301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.123221    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.123301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.123235    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.124474 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.124452    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:41:59.124568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.124543    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:41:59.125009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.124987    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:59.125072 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.125014    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr" pod "2d5314ae-9bd7-44c1-9345-73dddd0b6723" (UID: "2d5314ae-9bd7-44c1-9345-73dddd0b6723"). InnerVolumeSpecName "kube-api-access-b5btr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:59.224883 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.224819    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.224883 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.224858    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5btr\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-kube-api-access-b5btr\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.224883 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.224876    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.224883 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.224889    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5314ae-9bd7-44c1-9345-73dddd0b6723-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:59.504386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.504347    5718 generic.go:356] "Generic (PLEG): container finished" podID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerID="4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950" exitCode=0
Oct 31 01:41:59.504589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.504395    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"
Oct 31 01:41:59.504589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.504430    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" event={"ID":"2d5314ae-9bd7-44c1-9345-73dddd0b6723","Type":"ContainerDied","Data":"4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950"}
Oct 31 01:41:59.504589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.504486    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274" event={"ID":"2d5314ae-9bd7-44c1-9345-73dddd0b6723","Type":"ContainerDied","Data":"0ba0f06da4a28b8f21fffda65c94b9822fc480cc54743ca4e590f41a09d51af7"}
Oct 31 01:41:59.504589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.504531    5718 scope.go:122] "RemoveContainer" containerID="4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950"
Oct 31 01:41:59.515229 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.515214    5718 scope.go:122] "RemoveContainer" containerID="4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950"
Oct 31 01:41:59.515495 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:41:59.515478    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950\": container with ID starting with 4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950 not found: ID does not exist" containerID="4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950"
Oct 31 01:41:59.515565 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.515501    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950"} err="failed to get container status \"4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950\": rpc error: code = NotFound desc = could not find container \"4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950\": container with ID starting with 4efee5e9dd9343691f1a336b5cd7d7c89d400dbbce662a7e6b460603c8493950 not found: ID does not exist"
Oct 31 01:41:59.675157 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.675123    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"]
Oct 31 01:41:59.725684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.725634    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-7f274"]
Oct 31 01:41:59.967923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:41:59.967841    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" path="/var/lib/kubelet/pods/2d5314ae-9bd7-44c1-9345-73dddd0b6723/volumes"
Oct 31 01:42:02.962749 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:02.962712    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"]
Oct 31 01:42:02.963175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:02.962907    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" podUID="5b03c8b3-2538-4079-bfb7-49eeb795379a" containerName="discovery" containerID="cri-o://9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353" gracePeriod=30
Oct 31 01:42:03.230666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.230587    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:42:03.377052 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377009    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.377227 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377167    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.377227 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377204    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.377317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377260    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.377317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377291    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.377406 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377348    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.377552 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377532    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert\") pod \"5b03c8b3-2538-4079-bfb7-49eeb795379a\" (UID: \"5b03c8b3-2538-4079-bfb7-49eeb795379a\") "
Oct 31 01:42:03.378262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.377774    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:03.378262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.378150    5718 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-ca-configmap\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.380353 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.380320    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:03.380444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.380372    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:03.380444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.380386    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "kube-api-access-ws262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:03.380571 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.380556    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:03.380707 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.380644    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:03.381108 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.381073    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs" pod "5b03c8b3-2538-4079-bfb7-49eeb795379a" (UID: "5b03c8b3-2538-4079-bfb7-49eeb795379a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:03.479358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.479323    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.479358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.479349    5718 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-csr-dns-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.479358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.479359    5718 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-cacerts\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.479603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.479370    5718 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b03c8b3-2538-4079-bfb7-49eeb795379a-istio-kubeconfig\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.479603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.479380    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws262\" (UniqueName: \"kubernetes.io/projected/5b03c8b3-2538-4079-bfb7-49eeb795379a-kube-api-access-ws262\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.479603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.479391    5718 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b03c8b3-2538-4079-bfb7-49eeb795379a-local-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:03.527899 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.527860    5718 generic.go:356] "Generic (PLEG): container finished" podID="5b03c8b3-2538-4079-bfb7-49eeb795379a" containerID="9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353" exitCode=0
Oct 31 01:42:03.528104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.527923    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"
Oct 31 01:42:03.528104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.527952    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" event={"ID":"5b03c8b3-2538-4079-bfb7-49eeb795379a","Type":"ContainerDied","Data":"9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353"}
Oct 31 01:42:03.528104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.528011    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4" event={"ID":"5b03c8b3-2538-4079-bfb7-49eeb795379a","Type":"ContainerDied","Data":"4211f4b5c20e2afd9eeb8e16a2adea25fa2ed9091a179556d66457536289e7f7"}
Oct 31 01:42:03.528104 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.528032    5718 scope.go:122] "RemoveContainer" containerID="9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353"
Oct 31 01:42:03.538792 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.538774    5718 scope.go:122] "RemoveContainer" containerID="9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353"
Oct 31 01:42:03.539089 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:42:03.539069    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353\": container with ID starting with 9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353 not found: ID does not exist" containerID="9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353"
Oct 31 01:42:03.539160 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.539096    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353"} err="failed to get container status \"9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353\": rpc error: code = NotFound desc = could not find container \"9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353\": container with ID starting with 9a1c3ca5114c5e911330cf5102a5aeaeb61259bc41cc388c31b47c2b9b3d4353 not found: ID does not exist"
Oct 31 01:42:03.695045 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.695011    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"]
Oct 31 01:42:03.698594 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.698565    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f4lr4"]
Oct 31 01:42:03.967566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:03.967476    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b03c8b3-2538-4079-bfb7-49eeb795379a" path="/var/lib/kubelet/pods/5b03c8b3-2538-4079-bfb7-49eeb795379a/volumes"
Oct 31 01:42:07.923996 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:42:07.923949    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:42:31.304539 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.304500    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"]
Oct 31 01:42:31.305307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305288    5718 cpu_manager.go:414] "containerMap: removing container" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy"
Oct 31 01:42:31.305355 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305311    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy"
Oct 31 01:42:31.305389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305366    5718 cpu_manager.go:414] "containerMap: removing container" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy"
Oct 31 01:42:31.305389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305375    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy"
Oct 31 01:42:31.305450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305405    5718 cpu_manager.go:414] "containerMap: removing container" podUID="5b03c8b3-2538-4079-bfb7-49eeb795379a" containerName="discovery"
Oct 31 01:42:31.305450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305414    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b03c8b3-2538-4079-bfb7-49eeb795379a" containerName="discovery"
Oct 31 01:42:31.305561 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305547    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="f096d4aa-8b79-4d52-b5fd-bdb7ac6b2dd2" containerName="istio-proxy"
Oct 31 01:42:31.305606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305568    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b03c8b3-2538-4079-bfb7-49eeb795379a" containerName="discovery"
Oct 31 01:42:31.305606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.305585    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5314ae-9bd7-44c1-9345-73dddd0b6723" containerName="istio-proxy"
Oct 31 01:42:31.309486 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.309462    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.410311 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.410264    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.410508 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.410447    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.410603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.410584    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.410710 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.410679    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.413046 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.413026    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b2d49b-serving-cert\""
Oct 31 01:42:31.415361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.415339    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"]
Oct 31 01:42:31.511582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.511551    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.511819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.511694    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.511819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.511790    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.511911 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.511823    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.511911 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:42:31.511893    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-serving-cert: secret "maas-api-e2e-ipp-b-b2d49b-serving-cert" not found
Oct 31 01:42:31.511990 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:42:31.511976    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls podName:cc40539a-0207-4302-83d6-e958d63a76a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:42:32.011953413 +0000 UTC m=+2268.175096825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls") pod "maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" (UID: "cc40539a-0207-4302-83d6-e958d63a76a4") : secret "maas-api-e2e-ipp-b-b2d49b-serving-cert" not found
Oct 31 01:42:31.512298 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.512276    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.514384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.514368    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:31.572088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:31.572014    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:32.016737 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:32.016684    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:32.019389 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:32.019363    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:32.222828 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:32.222789    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:32.601849 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:32.601814    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"]
Oct 31 01:42:32.603667 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:42:32.603623    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc40539a_0207_4302_83d6_e958d63a76a4.slice/crio-004cf2f172d920b2e80ae6ebf7d5ab6042eaa16bde805df95f66d691925d15e1 WatchSource:0}: Error finding container 004cf2f172d920b2e80ae6ebf7d5ab6042eaa16bde805df95f66d691925d15e1: Status 404 returned error can't find the container with id 004cf2f172d920b2e80ae6ebf7d5ab6042eaa16bde805df95f66d691925d15e1
Oct 31 01:42:32.697912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:32.697870    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" event={"ID":"cc40539a-0207-4302-83d6-e958d63a76a4","Type":"ContainerStarted","Data":"004cf2f172d920b2e80ae6ebf7d5ab6042eaa16bde805df95f66d691925d15e1"}
Oct 31 01:42:33.705690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:33.705616    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" event={"ID":"cc40539a-0207-4302-83d6-e958d63a76a4","Type":"ContainerStarted","Data":"e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5"}
Oct 31 01:42:33.706167 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:33.705823    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:33.827101 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:33.826997    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" podStartSLOduration=1.928000103 podStartE2EDuration="2.826985253s" totalImagesPullingTime="898.98515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:33.826827894 +0000 UTC m=+2269.989971312" watchObservedRunningTime="2026-10-31 01:42:33.826985253 +0000 UTC m=+2269.990128667"
Oct 31 01:42:37.924718 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:42:37.924631    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:42:39.717804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:39.717774    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:42:43.557915 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.557875    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"]
Oct 31 01:42:43.561798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.561761    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.616573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.616540    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-df5d0b\""
Oct 31 01:42:43.653091 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.653058    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrn2s\" (UniqueName: \"kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.653283 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.653221    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.666015 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.665988    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-df5d0b-dockercfg-6khhw\""
Oct 31 01:42:43.668359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.668338    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"]
Oct 31 01:42:43.754567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.754530    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrn2s\" (UniqueName: \"kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.754829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.754636    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.755293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.755274    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.812338 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.812264    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrn2s\" (UniqueName: \"kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:43.875488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:43.875455    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:44.252966 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.252936    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"]
Oct 31 01:42:44.254545 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:42:44.254511    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313e8510_2fa5_4362_accf_f8f13bfabcb3.slice/crio-cdcf8a34ce017c1d5d6587fd3d1c5862eec0e8a4144c47b5957819cccc5e6911 WatchSource:0}: Error finding container cdcf8a34ce017c1d5d6587fd3d1c5862eec0e8a4144c47b5957819cccc5e6911: Status 404 returned error can't find the container with id cdcf8a34ce017c1d5d6587fd3d1c5862eec0e8a4144c47b5957819cccc5e6911
Oct 31 01:42:44.360198 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.360154    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"]
Oct 31 01:42:44.364292 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.364270    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.420115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.420087    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"]
Oct 31 01:42:44.462242 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.462203    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.462425 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.462254    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29j9b\" (UniqueName: \"kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b\") pod \"payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.563247 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.563156    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.563247 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.563227    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29j9b\" (UniqueName: \"kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b\") pod \"payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.563925 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.563906    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.620957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.620928    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29j9b\" (UniqueName: \"kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b\") pod \"payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.677589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.677551    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:44.777803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.776880    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh" event={"ID":"313e8510-2fa5-4362-accf-f8f13bfabcb3","Type":"ContainerStarted","Data":"33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961"}
Oct 31 01:42:44.777803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.776948    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh" event={"ID":"313e8510-2fa5-4362-accf-f8f13bfabcb3","Type":"ContainerStarted","Data":"cdcf8a34ce017c1d5d6587fd3d1c5862eec0e8a4144c47b5957819cccc5e6911"}
Oct 31 01:42:44.777803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.776982    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:44.896984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:44.896868    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh" podStartSLOduration=1.896850196 podStartE2EDuration="1.896850196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:44.896168515 +0000 UTC m=+2281.059311930" watchObservedRunningTime="2026-10-31 01:42:44.896850196 +0000 UTC m=+2281.059993612"
Oct 31 01:42:45.058853 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:45.058803    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"]
Oct 31 01:42:45.069262 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:42:45.069224    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ca9150_4878_4c9b_8216_d9d9bcebfce5.slice/crio-6077e42da8e8385cda0e7792ee0c41a1c38d04a2818a32622cad15a552f2ae9c WatchSource:0}: Error finding container 6077e42da8e8385cda0e7792ee0c41a1c38d04a2818a32622cad15a552f2ae9c: Status 404 returned error can't find the container with id 6077e42da8e8385cda0e7792ee0c41a1c38d04a2818a32622cad15a552f2ae9c
Oct 31 01:42:45.786682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:45.786125    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5" event={"ID":"18ca9150-4878-4c9b-8216-d9d9bcebfce5","Type":"ContainerStarted","Data":"ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b"}
Oct 31 01:42:45.786682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:45.786193    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5" event={"ID":"18ca9150-4878-4c9b-8216-d9d9bcebfce5","Type":"ContainerStarted","Data":"6077e42da8e8385cda0e7792ee0c41a1c38d04a2818a32622cad15a552f2ae9c"}
Oct 31 01:42:46.792248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:46.792211    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:42:55.789092 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:55.789061    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:42:55.914153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:55.914084    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5" podStartSLOduration=11.914066114 podStartE2EDuration="11.914066114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:45.903866247 +0000 UTC m=+2282.067009674" watchObservedRunningTime="2026-10-31 01:42:55.914066114 +0000 UTC m=+2292.077209531"
Oct 31 01:42:57.800748 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:42:57.800715    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:43:07.929212 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:43:07.929172    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:43:24.490258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:24.490217    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"]
Oct 31 01:43:24.491156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:24.490600    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy" containerID="cri-o://8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f" gracePeriod=30
Oct 31 01:43:29.750319 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.750294    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:43:29.913120 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913014    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913120 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913091    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913138    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913168    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913210    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913264    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913297    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913361    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913407    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:29.913703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913463    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token\") pod \"74a682ba-22d6-45a4-a9b9-713e5c59447a\" (UID: \"74a682ba-22d6-45a4-a9b9-713e5c59447a\") "
Oct 31 01:43:29.913703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913572    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:29.913703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913603    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:29.913703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913613    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:29.914120 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.913778    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:29.914209 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.914186    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:29.914269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.914215    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a682ba-22d6-45a4-a9b9-713e5c59447a-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:29.914269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.914234    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:29.914269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.914249    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:29.914269 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.914264    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:29.915694 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.915640    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:29.915694 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.915686    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "kube-api-access-xd9kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:29.915882 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.915719    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:29.915882 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:29.915818    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo" pod "74a682ba-22d6-45a4-a9b9-713e5c59447a" (UID: "74a682ba-22d6-45a4-a9b9-713e5c59447a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:43:30.015414 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.015382    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:30.015414 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.015410    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:30.015623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.015423    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a682ba-22d6-45a4-a9b9-713e5c59447a-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:30.015623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.015436    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd9kh\" (UniqueName: \"kubernetes.io/projected/74a682ba-22d6-45a4-a9b9-713e5c59447a-kube-api-access-xd9kh\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:30.052267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.052169    5718 generic.go:356] "Generic (PLEG): container finished" podID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerID="8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f" exitCode=0
Oct 31 01:43:30.052267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.052228    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"
Oct 31 01:43:30.052267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.052264    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" event={"ID":"74a682ba-22d6-45a4-a9b9-713e5c59447a","Type":"ContainerDied","Data":"8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f"}
Oct 31 01:43:30.052267 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.052355    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc" event={"ID":"74a682ba-22d6-45a4-a9b9-713e5c59447a","Type":"ContainerDied","Data":"6685c4417d47bd36ebc1dfa1c289fc0e6236f01424ec58073356f9d90fe31b75"}
Oct 31 01:43:30.052901 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.052418    5718 scope.go:122] "RemoveContainer" containerID="8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f"
Oct 31 01:43:30.065936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.065905    5718 scope.go:122] "RemoveContainer" containerID="8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f"
Oct 31 01:43:30.066362 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:43:30.066328    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f\": container with ID starting with 8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f not found: ID does not exist" containerID="8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f"
Oct 31 01:43:30.066498 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.066366    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f"} err="failed to get container status \"8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f\": rpc error: code = NotFound desc = could not find container \"8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f\": container with ID starting with 8c79b9b3722c28cf1fe2f1c6c33f5e68362f06acb16b220f1a797057d8725c3f not found: ID does not exist"
Oct 31 01:43:30.224340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.224252    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"]
Oct 31 01:43:30.226473 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:30.226451    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-44qpc"]
Oct 31 01:43:31.970591 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:31.970559    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" path="/var/lib/kubelet/pods/74a682ba-22d6-45a4-a9b9-713e5c59447a/volumes"
Oct 31 01:43:32.604132 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.604096    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"]
Oct 31 01:43:32.604715 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.604693    5718 cpu_manager.go:414] "containerMap: removing container" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy"
Oct 31 01:43:32.604715 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.604715    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy"
Oct 31 01:43:32.604908 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.604899    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a682ba-22d6-45a4-a9b9-713e5c59447a" containerName="istio-proxy"
Oct 31 01:43:32.609820 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.609804    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.650992 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.650942    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5vw\" (UniqueName: \"kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.651181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.651093    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.651181 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.651166    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.651275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.651222    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.666056 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.666027    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-71bc98-serving-cert\""
Oct 31 01:43:32.719154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.719114    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"]
Oct 31 01:43:32.752859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.752825    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5vw\" (UniqueName: \"kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.753069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.752916    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.753069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.752975    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.753069 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.753024    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.753859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.753822    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.756169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.756140    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.756298 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.756275    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.811750 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.811721    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5vw\" (UniqueName: \"kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw\") pod \"maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:32.923343 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:32.923256    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:33.312482 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:33.312457    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"]
Oct 31 01:43:33.313286 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:43:33.313253    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913908ff_b918_4b1a_a058_b862329dad9a.slice/crio-491c423d293d7e0dadae5d20b036c24a55d2a38bceeb67d402614fc09fa84e1b WatchSource:0}: Error finding container 491c423d293d7e0dadae5d20b036c24a55d2a38bceeb67d402614fc09fa84e1b: Status 404 returned error can't find the container with id 491c423d293d7e0dadae5d20b036c24a55d2a38bceeb67d402614fc09fa84e1b
Oct 31 01:43:34.079593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:34.079560    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz" event={"ID":"913908ff-b918-4b1a-a058-b862329dad9a","Type":"ContainerStarted","Data":"491c423d293d7e0dadae5d20b036c24a55d2a38bceeb67d402614fc09fa84e1b"}
Oct 31 01:43:35.087018 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:35.086984    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz" event={"ID":"913908ff-b918-4b1a-a058-b862329dad9a","Type":"ContainerStarted","Data":"ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3"}
Oct 31 01:43:35.087438 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:35.087052    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:35.209220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:35.209166    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz" podStartSLOduration=2.302487129 podStartE2EDuration="3.209150609s" totalImagesPullingTime="906.66348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:35.206848593 +0000 UTC m=+2331.369992008" watchObservedRunningTime="2026-10-31 01:43:35.209150609 +0000 UTC m=+2331.372294023"
Oct 31 01:43:37.924586 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:43:37.924548    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:43:41.099479 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:41.099445    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:43:54.513710 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:54.513615    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"]
Oct 31 01:43:54.514128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:54.513875    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5" podUID="18ca9150-4878-4c9b-8216-d9d9bcebfce5" containerName="payload-processing" containerID="cri-o://ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b" gracePeriod=30
Oct 31 01:43:54.566180 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:54.566151    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"]
Oct 31 01:43:54.566405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:54.566364    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh" podUID="313e8510-2fa5-4362-accf-f8f13bfabcb3" containerName="payload-pre-processing" containerID="cri-o://33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961" gracePeriod=30
Oct 31 01:43:54.839556 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:54.839530    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:43:54.868861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:54.868836    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:43:55.010794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.010755    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume\") pod \"313e8510-2fa5-4362-accf-f8f13bfabcb3\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") "
Oct 31 01:43:55.010975 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.010805    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s\" (UniqueName: \"kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s\") pod \"313e8510-2fa5-4362-accf-f8f13bfabcb3\" (UID: \"313e8510-2fa5-4362-accf-f8f13bfabcb3\") "
Oct 31 01:43:55.010975 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.010878    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume\") pod \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") "
Oct 31 01:43:55.010975 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.010909    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b\" (UniqueName: \"kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b\") pod \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\" (UID: \"18ca9150-4878-4c9b-8216-d9d9bcebfce5\") "
Oct 31 01:43:55.011279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.011251    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume" pod "313e8510-2fa5-4362-accf-f8f13bfabcb3" (UID: "313e8510-2fa5-4362-accf-f8f13bfabcb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:55.011373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.011253    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume" pod "18ca9150-4878-4c9b-8216-d9d9bcebfce5" (UID: "18ca9150-4878-4c9b-8216-d9d9bcebfce5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:55.011491 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.011473    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313e8510-2fa5-4362-accf-f8f13bfabcb3-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:55.011537 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.011497    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ca9150-4878-4c9b-8216-d9d9bcebfce5-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:55.013312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.013288    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s" pod "313e8510-2fa5-4362-accf-f8f13bfabcb3" (UID: "313e8510-2fa5-4362-accf-f8f13bfabcb3"). InnerVolumeSpecName "kube-api-access-qrn2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:55.013419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.013322    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b" pod "18ca9150-4878-4c9b-8216-d9d9bcebfce5" (UID: "18ca9150-4878-4c9b-8216-d9d9bcebfce5"). InnerVolumeSpecName "kube-api-access-29j9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:55.122407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.122321    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrn2s\" (UniqueName: \"kubernetes.io/projected/313e8510-2fa5-4362-accf-f8f13bfabcb3-kube-api-access-qrn2s\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:55.122714 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.122686    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29j9b\" (UniqueName: \"kubernetes.io/projected/18ca9150-4878-4c9b-8216-d9d9bcebfce5-kube-api-access-29j9b\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:55.202008 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.201966    5718 generic.go:356] "Generic (PLEG): container finished" podID="18ca9150-4878-4c9b-8216-d9d9bcebfce5" containerID="ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b" exitCode=0
Oct 31 01:43:55.202199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.202030    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"
Oct 31 01:43:55.202199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.202050    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5" event={"ID":"18ca9150-4878-4c9b-8216-d9d9bcebfce5","Type":"ContainerDied","Data":"ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b"}
Oct 31 01:43:55.202199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.202131    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5" event={"ID":"18ca9150-4878-4c9b-8216-d9d9bcebfce5","Type":"ContainerDied","Data":"6077e42da8e8385cda0e7792ee0c41a1c38d04a2818a32622cad15a552f2ae9c"}
Oct 31 01:43:55.202199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.202156    5718 scope.go:122] "RemoveContainer" containerID="ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b"
Oct 31 01:43:55.203742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.203714    5718 generic.go:356] "Generic (PLEG): container finished" podID="313e8510-2fa5-4362-accf-f8f13bfabcb3" containerID="33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961" exitCode=0
Oct 31 01:43:55.203880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.203775    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"
Oct 31 01:43:55.203880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.203786    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh" event={"ID":"313e8510-2fa5-4362-accf-f8f13bfabcb3","Type":"ContainerDied","Data":"33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961"}
Oct 31 01:43:55.203880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.203826    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh" event={"ID":"313e8510-2fa5-4362-accf-f8f13bfabcb3","Type":"ContainerDied","Data":"cdcf8a34ce017c1d5d6587fd3d1c5862eec0e8a4144c47b5957819cccc5e6911"}
Oct 31 01:43:55.213017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.213001    5718 scope.go:122] "RemoveContainer" containerID="ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b"
Oct 31 01:43:55.213278 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:43:55.213253    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b\": container with ID starting with ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b not found: ID does not exist" containerID="ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b"
Oct 31 01:43:55.213340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.213291    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b"} err="failed to get container status \"ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b\": rpc error: code = NotFound desc = could not find container \"ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b\": container with ID starting with ceaa640fd76906f3cac6588df7d08b2a738aa6f2a3b7acdfb6ea94fceaad964b not found: ID does not exist"
Oct 31 01:43:55.213340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.213318    5718 scope.go:122] "RemoveContainer" containerID="33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961"
Oct 31 01:43:55.222443 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.222421    5718 scope.go:122] "RemoveContainer" containerID="33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961"
Oct 31 01:43:55.222742 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:43:55.222720    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961\": container with ID starting with 33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961 not found: ID does not exist" containerID="33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961"
Oct 31 01:43:55.222800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.222750    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961"} err="failed to get container status \"33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961\": rpc error: code = NotFound desc = could not find container \"33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961\": container with ID starting with 33329bd1a5434013f364646c1e61e6a72bf2ffe36351d8facb850f8dabd16961 not found: ID does not exist"
Oct 31 01:43:55.367395 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.367359    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"]
Oct 31 01:43:55.371309 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.371285    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-f4fc9b794-c65p5"]
Oct 31 01:43:55.481951 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.481859    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"]
Oct 31 01:43:55.483938 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.483906    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-5755d869ff-kfxnh"]
Oct 31 01:43:55.967544 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.967507    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ca9150-4878-4c9b-8216-d9d9bcebfce5" path="/var/lib/kubelet/pods/18ca9150-4878-4c9b-8216-d9d9bcebfce5/volumes"
Oct 31 01:43:55.967978 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:43:55.967961    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313e8510-2fa5-4362-accf-f8f13bfabcb3" path="/var/lib/kubelet/pods/313e8510-2fa5-4362-accf-f8f13bfabcb3/volumes"
Oct 31 01:44:07.925129 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:44:07.925091    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:44:14.592684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:14.592608    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"]
Oct 31 01:44:14.593361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:14.592904    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy" containerID="cri-o://60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1" gracePeriod=30
Oct 31 01:44:19.859621 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:19.859557    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032283    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032391    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032486    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032528    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032555    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032605    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032707    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032763    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.033620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.032794    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy\") pod \"f7897809-3e59-4b95-97d5-619a3e00e9f4\" (UID: \"f7897809-3e59-4b95-97d5-619a3e00e9f4\") "
Oct 31 01:44:20.034300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.033777    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.034300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.033977    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.034703 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.034673    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.034864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.034740    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.035150 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.035110    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:20.036935 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.036908    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:20.037768 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.037741    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "kube-api-access-pkc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:20.038585 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.038558    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:20.039756 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.039734    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy" pod "f7897809-3e59-4b95-97d5-619a3e00e9f4" (UID: "f7897809-3e59-4b95-97d5-619a3e00e9f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.134779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134694    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.134779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134726    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.134779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134751    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7897809-3e59-4b95-97d5-619a3e00e9f4-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.134779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134765    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.134779 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134778    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.135166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134793    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.135166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134804    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.135166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134816    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7897809-3e59-4b95-97d5-619a3e00e9f4-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.135166 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.134829    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkc8w\" (UniqueName: \"kubernetes.io/projected/f7897809-3e59-4b95-97d5-619a3e00e9f4-kube-api-access-pkc8w\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.357735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.357701    5718 generic.go:356] "Generic (PLEG): container finished" podID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerID="60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1" exitCode=0
Oct 31 01:44:20.357912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.357754    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"
Oct 31 01:44:20.357912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.357770    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" event={"ID":"f7897809-3e59-4b95-97d5-619a3e00e9f4","Type":"ContainerDied","Data":"60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1"}
Oct 31 01:44:20.357912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.357816    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv" event={"ID":"f7897809-3e59-4b95-97d5-619a3e00e9f4","Type":"ContainerDied","Data":"9a39d2de6201086bf47a2d20de31543a946deb0320f954dfd69e0f51c466ef85"}
Oct 31 01:44:20.357912 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.357838    5718 scope.go:122] "RemoveContainer" containerID="60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1"
Oct 31 01:44:20.368920 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.368902    5718 scope.go:122] "RemoveContainer" containerID="60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1"
Oct 31 01:44:20.369197 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:44:20.369177    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1\": container with ID starting with 60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1 not found: ID does not exist" containerID="60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1"
Oct 31 01:44:20.369240 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.369206    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1"} err="failed to get container status \"60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1\": rpc error: code = NotFound desc = could not find container \"60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1\": container with ID starting with 60371e9884bf414ce6b131a8b9b88da63329e93a08e1c22aa1108d83394c58d1 not found: ID does not exist"
Oct 31 01:44:20.524132 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.524094    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"]
Oct 31 01:44:20.574812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:20.574777    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-zqjmv"]
Oct 31 01:44:21.967001 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:21.966971    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" path="/var/lib/kubelet/pods/f7897809-3e59-4b95-97d5-619a3e00e9f4/volumes"
Oct 31 01:44:37.923886 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:44:37.923839    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:44:42.820782 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:42.820752    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"]
Oct 31 01:44:42.823413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:42.821015    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz" podUID="913908ff-b918-4b1a-a058-b862329dad9a" containerName="maas-api" containerID="cri-o://ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3" gracePeriod=30
Oct 31 01:44:43.090838 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.090807    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:44:43.206287 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.206250    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls\") pod \"913908ff-b918-4b1a-a058-b862329dad9a\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") "
Oct 31 01:44:43.206484 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.206351    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw\" (UniqueName: \"kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw\") pod \"913908ff-b918-4b1a-a058-b862329dad9a\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") "
Oct 31 01:44:43.206550 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.206518    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca\") pod \"913908ff-b918-4b1a-a058-b862329dad9a\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") "
Oct 31 01:44:43.206550 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.206542    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs\") pod \"913908ff-b918-4b1a-a058-b862329dad9a\" (UID: \"913908ff-b918-4b1a-a058-b862329dad9a\") "
Oct 31 01:44:43.207248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.207153    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca" pod "913908ff-b918-4b1a-a058-b862329dad9a" (UID: "913908ff-b918-4b1a-a058-b862329dad9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:43.209209 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.209183    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls" pod "913908ff-b918-4b1a-a058-b862329dad9a" (UID: "913908ff-b918-4b1a-a058-b862329dad9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:43.209335 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.209308    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw" pod "913908ff-b918-4b1a-a058-b862329dad9a" (UID: "913908ff-b918-4b1a-a058-b862329dad9a"). InnerVolumeSpecName "kube-api-access-9j5vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:43.210889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.210862    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs" pod "913908ff-b918-4b1a-a058-b862329dad9a" (UID: "913908ff-b918-4b1a-a058-b862329dad9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:43.307666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.307620    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/913908ff-b918-4b1a-a058-b862329dad9a-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.307666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.307671    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.307666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.307684    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/913908ff-b918-4b1a-a058-b862329dad9a-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.307666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.307694    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5vw\" (UniqueName: \"kubernetes.io/projected/913908ff-b918-4b1a-a058-b862329dad9a-kube-api-access-9j5vw\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:43.496736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.496695    5718 generic.go:356] "Generic (PLEG): container finished" podID="913908ff-b918-4b1a-a058-b862329dad9a" containerID="ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3" exitCode=0
Oct 31 01:44:43.496922 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.496780    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"
Oct 31 01:44:43.496922 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.496781    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz" event={"ID":"913908ff-b918-4b1a-a058-b862329dad9a","Type":"ContainerDied","Data":"ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3"}
Oct 31 01:44:43.496922 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.496849    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz" event={"ID":"913908ff-b918-4b1a-a058-b862329dad9a","Type":"ContainerDied","Data":"491c423d293d7e0dadae5d20b036c24a55d2a38bceeb67d402614fc09fa84e1b"}
Oct 31 01:44:43.496922 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.496881    5718 scope.go:122] "RemoveContainer" containerID="ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3"
Oct 31 01:44:43.509460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.509440    5718 scope.go:122] "RemoveContainer" containerID="ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3"
Oct 31 01:44:43.509830 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:44:43.509801    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3\": container with ID starting with ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3 not found: ID does not exist" containerID="ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3"
Oct 31 01:44:43.509929 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.509845    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3"} err="failed to get container status \"ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3\": rpc error: code = NotFound desc = could not find container \"ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3\": container with ID starting with ef9375a92d93669c27f5d117e1811c739df73cd104ba908b66b6fb596efc71f3 not found: ID does not exist"
Oct 31 01:44:43.667128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.667076    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"]
Oct 31 01:44:43.718985 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.718945    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-57d96b484c-rxzzz"]
Oct 31 01:44:43.724530 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.724499    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"]
Oct 31 01:44:43.725101 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.724771    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" podUID="9fed430c-ca33-4e1a-b778-730f97456126" containerName="maas-api" containerID="cri-o://9d0564f298141a1f179e040c3427ae3160f8e750184cc6be7f66deb9781e4bd0" gracePeriod=30
Oct 31 01:44:43.758972 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:44:43.758892    5718 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fed430c_ca33_4e1a_b778_730f97456126.slice/crio-9d0564f298141a1f179e040c3427ae3160f8e750184cc6be7f66deb9781e4bd0.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:44:43.970614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:43.970577    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913908ff-b918-4b1a-a058-b862329dad9a" path="/var/lib/kubelet/pods/913908ff-b918-4b1a-a058-b862329dad9a/volumes"
Oct 31 01:44:44.033857 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.033828    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:44:44.192841 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.192800    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"]
Oct 31 01:44:44.193086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.193044    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx" podUID="507dfb77-eaae-4de5-991b-2bf8607e3475" containerName="payload-processing" containerID="cri-o://34b36d736c1958ccb9010e2c6e9cfbc85be5df468c8b4209a2021f438df18398" gracePeriod=30
Oct 31 01:44:44.219139 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.219113    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\") pod \"9fed430c-ca33-4e1a-b778-730f97456126\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") "
Oct 31 01:44:44.219257 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.219239    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca\") pod \"9fed430c-ca33-4e1a-b778-730f97456126\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") "
Oct 31 01:44:44.219331 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.219316    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84\" (UniqueName: \"kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84\") pod \"9fed430c-ca33-4e1a-b778-730f97456126\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") "
Oct 31 01:44:44.219370 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.219351    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs\") pod \"9fed430c-ca33-4e1a-b778-730f97456126\" (UID: \"9fed430c-ca33-4e1a-b778-730f97456126\") "
Oct 31 01:44:44.219640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.219616    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca" pod "9fed430c-ca33-4e1a-b778-730f97456126" (UID: "9fed430c-ca33-4e1a-b778-730f97456126"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:44.219851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.219827    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fed430c-ca33-4e1a-b778-730f97456126-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.221682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.221632    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs" pod "9fed430c-ca33-4e1a-b778-730f97456126" (UID: "9fed430c-ca33-4e1a-b778-730f97456126"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:44.221957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.221936    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls" pod "9fed430c-ca33-4e1a-b778-730f97456126" (UID: "9fed430c-ca33-4e1a-b778-730f97456126"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:44.222014 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.221952    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84" pod "9fed430c-ca33-4e1a-b778-730f97456126" (UID: "9fed430c-ca33-4e1a-b778-730f97456126"). InnerVolumeSpecName "kube-api-access-cwb84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:44.256426 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.256392    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"]
Oct 31 01:44:44.256715 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.256646    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl" podUID="901612dd-b099-4d9b-b410-f9c465dd5798" containerName="payload-pre-processing" containerID="cri-o://e6d8755462f37cdf9f85a2246216752d75a2566d65fd74e8dc09388ac826f920" gracePeriod=30
Oct 31 01:44:44.322584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.322539    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwb84\" (UniqueName: \"kubernetes.io/projected/9fed430c-ca33-4e1a-b778-730f97456126-kube-api-access-cwb84\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.322584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.322575    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.322584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.322590    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fed430c-ca33-4e1a-b778-730f97456126-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.412147 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.412109    5718 scope.go:122] "RemoveContainer" containerID="9d0564f298141a1f179e040c3427ae3160f8e750184cc6be7f66deb9781e4bd0"
Oct 31 01:44:44.507050 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.507009    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" event={"ID":"9fed430c-ca33-4e1a-b778-730f97456126","Type":"ContainerDied","Data":"9d0564f298141a1f179e040c3427ae3160f8e750184cc6be7f66deb9781e4bd0"}
Oct 31 01:44:44.507187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.507083    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6" event={"ID":"9fed430c-ca33-4e1a-b778-730f97456126","Type":"ContainerDied","Data":"9d6569e9eede3fcca4e76fb9669c2000ddde164a90e8437e2262182d68578511"}
Oct 31 01:44:44.510471 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.509640    5718 generic.go:356] "Generic (PLEG): container finished" podID="901612dd-b099-4d9b-b410-f9c465dd5798" containerID="e6d8755462f37cdf9f85a2246216752d75a2566d65fd74e8dc09388ac826f920" exitCode=0
Oct 31 01:44:44.510471 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.509756    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl" event={"ID":"901612dd-b099-4d9b-b410-f9c465dd5798","Type":"ContainerDied","Data":"e6d8755462f37cdf9f85a2246216752d75a2566d65fd74e8dc09388ac826f920"}
Oct 31 01:44:44.513526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.513495    5718 generic.go:356] "Generic (PLEG): container finished" podID="507dfb77-eaae-4de5-991b-2bf8607e3475" containerID="34b36d736c1958ccb9010e2c6e9cfbc85be5df468c8b4209a2021f438df18398" exitCode=0
Oct 31 01:44:44.513694 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.513603    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"
Oct 31 01:44:44.528748 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.516031    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx" event={"ID":"507dfb77-eaae-4de5-991b-2bf8607e3475","Type":"ContainerDied","Data":"34b36d736c1958ccb9010e2c6e9cfbc85be5df468c8b4209a2021f438df18398"}
Oct 31 01:44:44.541891 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.541858    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:44:44.570763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.570733    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:44:44.726578 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.726529    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume\") pod \"507dfb77-eaae-4de5-991b-2bf8607e3475\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") "
Oct 31 01:44:44.726850 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.726632    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52\" (UniqueName: \"kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52\") pod \"507dfb77-eaae-4de5-991b-2bf8607e3475\" (UID: \"507dfb77-eaae-4de5-991b-2bf8607e3475\") "
Oct 31 01:44:44.726850 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.726696    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume\") pod \"901612dd-b099-4d9b-b410-f9c465dd5798\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") "
Oct 31 01:44:44.726850 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.726721    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk\" (UniqueName: \"kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk\") pod \"901612dd-b099-4d9b-b410-f9c465dd5798\" (UID: \"901612dd-b099-4d9b-b410-f9c465dd5798\") "
Oct 31 01:44:44.727695 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.727624    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume" pod "507dfb77-eaae-4de5-991b-2bf8607e3475" (UID: "507dfb77-eaae-4de5-991b-2bf8607e3475"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:44.727926 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.727901    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume" pod "901612dd-b099-4d9b-b410-f9c465dd5798" (UID: "901612dd-b099-4d9b-b410-f9c465dd5798"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:44.731822 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.731788    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk" pod "901612dd-b099-4d9b-b410-f9c465dd5798" (UID: "901612dd-b099-4d9b-b410-f9c465dd5798"). InnerVolumeSpecName "kube-api-access-5svkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:44.731952 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.731832    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52" pod "507dfb77-eaae-4de5-991b-2bf8607e3475" (UID: "507dfb77-eaae-4de5-991b-2bf8607e3475"). InnerVolumeSpecName "kube-api-access-j8d52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:44.752366 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.752321    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"]
Oct 31 01:44:44.754213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.754188    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-447154-7c9fb8c869-frsr6"]
Oct 31 01:44:44.828771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.828674    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507dfb77-eaae-4de5-991b-2bf8607e3475-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.828771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.828704    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8d52\" (UniqueName: \"kubernetes.io/projected/507dfb77-eaae-4de5-991b-2bf8607e3475-kube-api-access-j8d52\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.828771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.828719    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/901612dd-b099-4d9b-b410-f9c465dd5798-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:44.828771 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:44.828771    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5svkk\" (UniqueName: \"kubernetes.io/projected/901612dd-b099-4d9b-b410-f9c465dd5798-kube-api-access-5svkk\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:45.521626 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.521592    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"
Oct 31 01:44:45.522113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.521596    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl" event={"ID":"901612dd-b099-4d9b-b410-f9c465dd5798","Type":"ContainerDied","Data":"c26a85eeae92296cf13d0981157baf3bb3e2412c460bc367f4238b15c3194b7e"}
Oct 31 01:44:45.522113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.521734    5718 scope.go:122] "RemoveContainer" containerID="e6d8755462f37cdf9f85a2246216752d75a2566d65fd74e8dc09388ac826f920"
Oct 31 01:44:45.525957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.525926    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx" event={"ID":"507dfb77-eaae-4de5-991b-2bf8607e3475","Type":"ContainerDied","Data":"7a869d6426f02cddf3da19af2e4c238362a7b6d5f8843b4d78ef70ce20387dd8"}
Oct 31 01:44:45.526065 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.525943    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"
Oct 31 01:44:45.533468 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.533449    5718 scope.go:122] "RemoveContainer" containerID="34b36d736c1958ccb9010e2c6e9cfbc85be5df468c8b4209a2021f438df18398"
Oct 31 01:44:45.686844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.686809    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"]
Oct 31 01:44:45.692693 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.692647    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-86f756f86b-vqtbx"]
Oct 31 01:44:45.803496 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.803413    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"]
Oct 31 01:44:45.806586 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.806562    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-54c6dc-bbxpl"]
Oct 31 01:44:45.967728 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.967696    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507dfb77-eaae-4de5-991b-2bf8607e3475" path="/var/lib/kubelet/pods/507dfb77-eaae-4de5-991b-2bf8607e3475/volumes"
Oct 31 01:44:45.968095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.968082    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901612dd-b099-4d9b-b410-f9c465dd5798" path="/var/lib/kubelet/pods/901612dd-b099-4d9b-b410-f9c465dd5798/volumes"
Oct 31 01:44:45.968505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:45.968491    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fed430c-ca33-4e1a-b778-730f97456126" path="/var/lib/kubelet/pods/9fed430c-ca33-4e1a-b778-730f97456126/volumes"
Oct 31 01:44:51.639994 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.639954    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"]
Oct 31 01:44:51.640562 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640513    5718 cpu_manager.go:414] "containerMap: removing container" podUID="507dfb77-eaae-4de5-991b-2bf8607e3475" containerName="payload-processing"
Oct 31 01:44:51.640562 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640532    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507dfb77-eaae-4de5-991b-2bf8607e3475" containerName="payload-processing"
Oct 31 01:44:51.640760 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640568    5718 cpu_manager.go:414] "containerMap: removing container" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy"
Oct 31 01:44:51.640760 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640577    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy"
Oct 31 01:44:51.640950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640931    5718 cpu_manager.go:414] "containerMap: removing container" podUID="913908ff-b918-4b1a-a058-b862329dad9a" containerName="maas-api"
Oct 31 01:44:51.641029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640950    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913908ff-b918-4b1a-a058-b862329dad9a" containerName="maas-api"
Oct 31 01:44:51.641029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640970    5718 cpu_manager.go:414] "containerMap: removing container" podUID="18ca9150-4878-4c9b-8216-d9d9bcebfce5" containerName="payload-processing"
Oct 31 01:44:51.641029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.640977    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ca9150-4878-4c9b-8216-d9d9bcebfce5" containerName="payload-processing"
Oct 31 01:44:51.641029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641011    5718 cpu_manager.go:414] "containerMap: removing container" podUID="901612dd-b099-4d9b-b410-f9c465dd5798" containerName="payload-pre-processing"
Oct 31 01:44:51.641029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641018    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901612dd-b099-4d9b-b410-f9c465dd5798" containerName="payload-pre-processing"
Oct 31 01:44:51.641029 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641028    5718 cpu_manager.go:414] "containerMap: removing container" podUID="313e8510-2fa5-4362-accf-f8f13bfabcb3" containerName="payload-pre-processing"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641036    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313e8510-2fa5-4362-accf-f8f13bfabcb3" containerName="payload-pre-processing"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641057    5718 cpu_manager.go:414] "containerMap: removing container" podUID="9fed430c-ca33-4e1a-b778-730f97456126" containerName="maas-api"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641064    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fed430c-ca33-4e1a-b778-730f97456126" containerName="maas-api"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641206    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="901612dd-b099-4d9b-b410-f9c465dd5798" containerName="payload-pre-processing"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641218    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="313e8510-2fa5-4362-accf-f8f13bfabcb3" containerName="payload-pre-processing"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641228    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fed430c-ca33-4e1a-b778-730f97456126" containerName="maas-api"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641234    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7897809-3e59-4b95-97d5-619a3e00e9f4" containerName="istio-proxy"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641242    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="913908ff-b918-4b1a-a058-b862329dad9a" containerName="maas-api"
Oct 31 01:44:51.641248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641248    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ca9150-4878-4c9b-8216-d9d9bcebfce5" containerName="payload-processing"
Oct 31 01:44:51.641547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.641256    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="507dfb77-eaae-4de5-991b-2bf8607e3475" containerName="payload-processing"
Oct 31 01:44:51.646003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.645984    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.700851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.700817    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"]
Oct 31 01:44:51.707700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.707646    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.707950 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.707921    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbkh\" (UniqueName: \"kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.748546 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.748516    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2d49b-dockercfg-tsrmm\""
Oct 31 01:44:51.748741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.748554    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2d49b\""
Oct 31 01:44:51.808965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.808927    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.809153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.809041    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbkh\" (UniqueName: \"kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.809664 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.809627    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.873418 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.873386    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbkh\" (UniqueName: \"kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:51.960900 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:51.960821    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:52.336794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:52.336452    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"]
Oct 31 01:44:52.339064 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:44:52.339037    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce34f90_44db_410b_907c_51433f1b5a49.slice/crio-203793cfedb8b7618ec00b9f23a4ef6f4a4a5e982243661760d03c6dc41d1336 WatchSource:0}: Error finding container 203793cfedb8b7618ec00b9f23a4ef6f4a4a5e982243661760d03c6dc41d1336: Status 404 returned error can't find the container with id 203793cfedb8b7618ec00b9f23a4ef6f4a4a5e982243661760d03c6dc41d1336
Oct 31 01:44:52.567720 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:52.567685    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw" event={"ID":"0ce34f90-44db-410b-907c-51433f1b5a49","Type":"ContainerStarted","Data":"a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f"}
Oct 31 01:44:52.567890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:52.567749    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw" event={"ID":"0ce34f90-44db-410b-907c-51433f1b5a49","Type":"ContainerStarted","Data":"203793cfedb8b7618ec00b9f23a4ef6f4a4a5e982243661760d03c6dc41d1336"}
Oct 31 01:44:52.567890 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:52.567807    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:44:52.684162 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:44:52.684047    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw" podStartSLOduration=1.684032603 podStartE2EDuration="1.684032603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:52.682717808 +0000 UTC m=+2408.845861224" watchObservedRunningTime="2026-10-31 01:44:52.684032603 +0000 UTC m=+2408.847176061"
Oct 31 01:45:00.218582 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.218543    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"]
Oct 31 01:45:00.222431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.222409    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.262137 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.262098    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"]
Oct 31 01:45:00.265904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.265887    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.286807 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.286773    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"]
Oct 31 01:45:00.290947 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.290890    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.309624 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.309594    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.309805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.309697    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.309805 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.309769    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5dz\" (UniqueName: \"kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.309975 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.309953    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l57g\" (UniqueName: \"kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.313113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.313083    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"]
Oct 31 01:45:00.315302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.315275    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"]
Oct 31 01:45:00.319049 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.319029    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.334337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.334309    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"]
Oct 31 01:45:00.338401 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.338373    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.367623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.367587    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"]
Oct 31 01:45:00.367814 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.367726    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:45:00.398266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.398211    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"]
Oct 31 01:45:00.409695 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.408950    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"]
Oct 31 01:45:00.421862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.412061    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8sm\" (UniqueName: \"kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.421862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.412180    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-53af08\"/\"kube-root-ca.crt\""
Oct 31 01:45:00.421862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.412176    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.421862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.412268    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-53af08\"/\"openshift-service-ca.crt\""
Oct 31 01:45:00.421862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.418272    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"]
Oct 31 01:45:00.421862 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.421120    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.422865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.422838    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890185-hqwjp\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.422984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.422883    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.422984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.422937    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l57g\" (UniqueName: \"kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.422984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.422967    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.423085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.422995    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8wz\" (UniqueName: \"kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.423085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.423031    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.423085 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.423053    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5dz\" (UniqueName: \"kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.423175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.423083    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.423175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.423117    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.423175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.423134    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77lg5\" (UniqueName: \"kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5\") pod \"maas-api-key-cleanup-29890185-hqwjp\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.423175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.423161    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.424142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.424121    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.424604 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.424588    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.468132 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.468091    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-53af08\"/\"default-dockercfg-fqf2v\""
Oct 31 01:45:00.468314 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.468171    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-53af08\"/\"ipp-route-53af08-kserve-self-signed-certs\""
Oct 31 01:45:00.488123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.488028    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5dz\" (UniqueName: \"kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.488123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.488067    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l57g\" (UniqueName: \"kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.524296 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524258    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.524513 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524320    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890185-hqwjp\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.524513 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524346    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.524513 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524394    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8wz\" (UniqueName: \"kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.524513 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524429    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.524809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524612    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.524809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524644    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77lg5\" (UniqueName: \"kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5\") pod \"maas-api-key-cleanup-29890185-hqwjp\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.524809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524696    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.524809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524730    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8sm\" (UniqueName: \"kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.524809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.524802    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.525250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.525225    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.525375 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.525267    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.525435 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.525376    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.525435 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.525392    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.526105 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.526062    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890185-hqwjp\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.527560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.527534    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.527899 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.527882    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.536590 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.536561    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:00.581015 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.580450    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:00.586326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.584785    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77lg5\" (UniqueName: \"kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5\") pod \"maas-api-key-cleanup-29890185-hqwjp\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.586836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.586386    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8sm\" (UniqueName: \"kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm\") pod \"ipp-route-53af08-kserve-cd76b4477-h64wj\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") " pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.586836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.586781    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8wz\" (UniqueName: \"kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.608560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.608366    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:00.638375 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.637895    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:00.653091 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.652899    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:00.940547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:00.940514    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"]
Oct 31 01:45:00.941543 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:00.941511    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df82c44_b352_4aad_9359_2cff6d73054f.slice/crio-529a37bffba4d2ba358afaaa743f1583455bd5e1bd0dc7003064d225e565582b WatchSource:0}: Error finding container 529a37bffba4d2ba358afaaa743f1583455bd5e1bd0dc7003064d225e565582b: Status 404 returned error can't find the container with id 529a37bffba4d2ba358afaaa743f1583455bd5e1bd0dc7003064d225e565582b
Oct 31 01:45:01.003547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.003519    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"]
Oct 31 01:45:01.004733 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:01.004705    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f80574_077d_485b_a6bb_78874aaee118.slice/crio-db74ccec60c2abbf9fc9e5915702d552d136eb58ac47267ca3f87672df565666 WatchSource:0}: Error finding container db74ccec60c2abbf9fc9e5915702d552d136eb58ac47267ca3f87672df565666: Status 404 returned error can't find the container with id db74ccec60c2abbf9fc9e5915702d552d136eb58ac47267ca3f87672df565666
Oct 31 01:45:01.127407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.127379    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"]
Oct 31 01:45:01.129225 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:01.129193    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ae1c4a_a518_4c08_9543_cb5ead10f9f0.slice/crio-b69dd513a02cc3210030230140effeae7d5329f46f690458de245132bcc7d4b6 WatchSource:0}: Error finding container b69dd513a02cc3210030230140effeae7d5329f46f690458de245132bcc7d4b6: Status 404 returned error can't find the container with id b69dd513a02cc3210030230140effeae7d5329f46f690458de245132bcc7d4b6
Oct 31 01:45:01.195778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.195702    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"]
Oct 31 01:45:01.196950 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:01.196921    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066229e9_7c64_4c3e_8e84_cae318086c20.slice/crio-9e50d269bb96cf12e5a7c8731ce1a229afdbdd98292bed031f113d38d699dfd3 WatchSource:0}: Error finding container 9e50d269bb96cf12e5a7c8731ce1a229afdbdd98292bed031f113d38d699dfd3: Status 404 returned error can't find the container with id 9e50d269bb96cf12e5a7c8731ce1a229afdbdd98292bed031f113d38d699dfd3
Oct 31 01:45:01.198231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.198203    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"]
Oct 31 01:45:01.200604 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:01.200581    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99412668_a12c_4264_b97c_38166feb93ab.slice/crio-95c4f9ab84d20c6fe2a21dba4f031be356205a4f406175173db46507456c36fc WatchSource:0}: Error finding container 95c4f9ab84d20c6fe2a21dba4f031be356205a4f406175173db46507456c36fc: Status 404 returned error can't find the container with id 95c4f9ab84d20c6fe2a21dba4f031be356205a4f406175173db46507456c36fc
Oct 31 01:45:01.648254 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.648214    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerStarted","Data":"47f45f30ae4d0354abd027719e7b72f06a500c55ebe3d615f9bef8ea7df66c2f"}
Oct 31 01:45:01.648790 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.648279    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerStarted","Data":"db74ccec60c2abbf9fc9e5915702d552d136eb58ac47267ca3f87672df565666"}
Oct 31 01:45:01.650993 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.650958    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerStarted","Data":"03856c31dd53e634e0847e2a8cdb0d0d4c91566e2029cc858389b5bb0efe21ef"}
Oct 31 01:45:01.651149 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.651018    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerStarted","Data":"9e50d269bb96cf12e5a7c8731ce1a229afdbdd98292bed031f113d38d699dfd3"}
Oct 31 01:45:01.652572 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.652545    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj" event={"ID":"99412668-a12c-4264-b97c-38166feb93ab","Type":"ContainerStarted","Data":"0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c"}
Oct 31 01:45:01.652731 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.652578    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj" event={"ID":"99412668-a12c-4264-b97c-38166feb93ab","Type":"ContainerStarted","Data":"95c4f9ab84d20c6fe2a21dba4f031be356205a4f406175173db46507456c36fc"}
Oct 31 01:45:01.653334 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.653313    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:01.654308 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.654282    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerStarted","Data":"f522ca017134dd88600573b3cfb64f6a8840b819c1e8540b38abe4e051ece2f7"}
Oct 31 01:45:01.654419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.654329    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerStarted","Data":"529a37bffba4d2ba358afaaa743f1583455bd5e1bd0dc7003064d225e565582b"}
Oct 31 01:45:01.656004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.655984    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerStarted","Data":"34a0d45e8dfca085066991c1bcdf84f4ef62783e328932d4d85becea189b718a"}
Oct 31 01:45:01.656082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.656026    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerStarted","Data":"b69dd513a02cc3210030230140effeae7d5329f46f690458de245132bcc7d4b6"}
Oct 31 01:45:01.764709 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.764630    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" podStartSLOduration=1.764614743 podStartE2EDuration="1.764614743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:01.763254269 +0000 UTC m=+2417.926397685" watchObservedRunningTime="2026-10-31 01:45:01.764614743 +0000 UTC m=+2417.927758158"
Oct 31 01:45:01.880514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.880456    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj" podStartSLOduration=1.880441975 podStartE2EDuration="1.880441975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:01.878187454 +0000 UTC m=+2418.041330871" watchObservedRunningTime="2026-10-31 01:45:01.880441975 +0000 UTC m=+2418.043585391"
Oct 31 01:45:01.993859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:01.993783    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" podStartSLOduration=1.993762449 podStartE2EDuration="1.993762449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:01.992619245 +0000 UTC m=+2418.155762663" watchObservedRunningTime="2026-10-31 01:45:01.993762449 +0000 UTC m=+2418.156905865"
Oct 31 01:45:02.110339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.110287    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" podStartSLOduration=2.110272914 podStartE2EDuration="2.110272914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.108671607 +0000 UTC m=+2418.271815013" watchObservedRunningTime="2026-10-31 01:45:02.110272914 +0000 UTC m=+2418.273416716"
Oct 31 01:45:02.225707 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.225587    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" podStartSLOduration=2.225568144 podStartE2EDuration="2.225568144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.223674591 +0000 UTC m=+2418.386818004" watchObservedRunningTime="2026-10-31 01:45:02.225568144 +0000 UTC m=+2418.388711562"
Oct 31 01:45:02.664153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.664116    5718 generic.go:356] "Generic (PLEG): container finished" podID="066229e9-7c64-4c3e-8e84-cae318086c20" containerID="03856c31dd53e634e0847e2a8cdb0d0d4c91566e2029cc858389b5bb0efe21ef" exitCode=60
Oct 31 01:45:02.664687 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.664201    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerDied","Data":"03856c31dd53e634e0847e2a8cdb0d0d4c91566e2029cc858389b5bb0efe21ef"}
Oct 31 01:45:02.664687 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.664433    5718 scope.go:122] "RemoveContainer" containerID="03856c31dd53e634e0847e2a8cdb0d0d4c91566e2029cc858389b5bb0efe21ef"
Oct 31 01:45:02.666036 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.666001    5718 generic.go:356] "Generic (PLEG): container finished" podID="4df82c44-b352-4aad-9359-2cff6d73054f" containerID="f522ca017134dd88600573b3cfb64f6a8840b819c1e8540b38abe4e051ece2f7" exitCode=60
Oct 31 01:45:02.666161 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.666054    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerDied","Data":"f522ca017134dd88600573b3cfb64f6a8840b819c1e8540b38abe4e051ece2f7"}
Oct 31 01:45:02.666345 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.666329    5718 scope.go:122] "RemoveContainer" containerID="f522ca017134dd88600573b3cfb64f6a8840b819c1e8540b38abe4e051ece2f7"
Oct 31 01:45:02.667996 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.667956    5718 generic.go:356] "Generic (PLEG): container finished" podID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerID="34a0d45e8dfca085066991c1bcdf84f4ef62783e328932d4d85becea189b718a" exitCode=60
Oct 31 01:45:02.668064 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.668041    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerDied","Data":"34a0d45e8dfca085066991c1bcdf84f4ef62783e328932d4d85becea189b718a"}
Oct 31 01:45:02.668338 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.668321    5718 scope.go:122] "RemoveContainer" containerID="34a0d45e8dfca085066991c1bcdf84f4ef62783e328932d4d85becea189b718a"
Oct 31 01:45:02.669924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.669903    5718 generic.go:356] "Generic (PLEG): container finished" podID="65f80574-077d-485b-a6bb-78874aaee118" containerID="47f45f30ae4d0354abd027719e7b72f06a500c55ebe3d615f9bef8ea7df66c2f" exitCode=60
Oct 31 01:45:02.670022 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.669931    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerDied","Data":"47f45f30ae4d0354abd027719e7b72f06a500c55ebe3d615f9bef8ea7df66c2f"}
Oct 31 01:45:02.670196 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:02.670181    5718 scope.go:122] "RemoveContainer" containerID="47f45f30ae4d0354abd027719e7b72f06a500c55ebe3d615f9bef8ea7df66c2f"
Oct 31 01:45:03.576729 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.576695    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:45:03.677829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.677800    5718 generic.go:356] "Generic (PLEG): container finished" podID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerID="795d0b6f37fe23331d7ccc9f4f8e43ad4c7b91059f93cebe5ecfea60280f82e5" exitCode=60
Oct 31 01:45:03.678273 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.677884    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerDied","Data":"795d0b6f37fe23331d7ccc9f4f8e43ad4c7b91059f93cebe5ecfea60280f82e5"}
Oct 31 01:45:03.678273 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.677957    5718 scope.go:122] "RemoveContainer" containerID="34a0d45e8dfca085066991c1bcdf84f4ef62783e328932d4d85becea189b718a"
Oct 31 01:45:03.678273 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.678232    5718 scope.go:122] "RemoveContainer" containerID="795d0b6f37fe23331d7ccc9f4f8e43ad4c7b91059f93cebe5ecfea60280f82e5"
Oct 31 01:45:03.678559 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:03.678535    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv_odh-ai-gateway-infra(46ae1c4a-a518-4c08-9543-cb5ead10f9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0"
Oct 31 01:45:03.680218 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.680193    5718 generic.go:356] "Generic (PLEG): container finished" podID="65f80574-077d-485b-a6bb-78874aaee118" containerID="0de341bb05366ae5caab03265b15f9f40be887bf7b970ef7bf6972df454fd4bd" exitCode=60
Oct 31 01:45:03.680314 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.680265    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerDied","Data":"0de341bb05366ae5caab03265b15f9f40be887bf7b970ef7bf6972df454fd4bd"}
Oct 31 01:45:03.680547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.680532    5718 scope.go:122] "RemoveContainer" containerID="0de341bb05366ae5caab03265b15f9f40be887bf7b970ef7bf6972df454fd4bd"
Oct 31 01:45:03.680839 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:03.680818    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b_odh-ai-gateway-infra(65f80574-077d-485b-a6bb-78874aaee118)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" podUID="65f80574-077d-485b-a6bb-78874aaee118"
Oct 31 01:45:03.682259 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.682236    5718 generic.go:356] "Generic (PLEG): container finished" podID="066229e9-7c64-4c3e-8e84-cae318086c20" containerID="e68220675328de16c197e21595c535c385b803d80f137adafe07433b92289602" exitCode=60
Oct 31 01:45:03.682357 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.682327    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerDied","Data":"e68220675328de16c197e21595c535c385b803d80f137adafe07433b92289602"}
Oct 31 01:45:03.682561 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.682546    5718 scope.go:122] "RemoveContainer" containerID="e68220675328de16c197e21595c535c385b803d80f137adafe07433b92289602"
Oct 31 01:45:03.682845 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:03.682820    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890185-hqwjp_odh-ai-gateway-infra(066229e9-7c64-4c3e-8e84-cae318086c20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" podUID="066229e9-7c64-4c3e-8e84-cae318086c20"
Oct 31 01:45:03.684189 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.684164    5718 generic.go:356] "Generic (PLEG): container finished" podID="4df82c44-b352-4aad-9359-2cff6d73054f" containerID="f3afd7c9c71c08a2aab923e4c3961c01766ae595b05d528f8e93a67b9d21aaff" exitCode=60
Oct 31 01:45:03.684302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.684250    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerDied","Data":"f3afd7c9c71c08a2aab923e4c3961c01766ae595b05d528f8e93a67b9d21aaff"}
Oct 31 01:45:03.684419 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.684407    5718 scope.go:122] "RemoveContainer" containerID="f3afd7c9c71c08a2aab923e4c3961c01766ae595b05d528f8e93a67b9d21aaff"
Oct 31 01:45:03.684619 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:03.684604    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph_odh-ai-gateway-infra(4df82c44-b352-4aad-9359-2cff6d73054f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" podUID="4df82c44-b352-4aad-9359-2cff6d73054f"
Oct 31 01:45:03.760347 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.760301    5718 scope.go:122] "RemoveContainer" containerID="47f45f30ae4d0354abd027719e7b72f06a500c55ebe3d615f9bef8ea7df66c2f"
Oct 31 01:45:03.778447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.778421    5718 scope.go:122] "RemoveContainer" containerID="03856c31dd53e634e0847e2a8cdb0d0d4c91566e2029cc858389b5bb0efe21ef"
Oct 31 01:45:03.791534 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:03.791511    5718 scope.go:122] "RemoveContainer" containerID="f522ca017134dd88600573b3cfb64f6a8840b819c1e8540b38abe4e051ece2f7"
Oct 31 01:45:04.693494 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:04.693464    5718 scope.go:122] "RemoveContainer" containerID="f3afd7c9c71c08a2aab923e4c3961c01766ae595b05d528f8e93a67b9d21aaff"
Oct 31 01:45:04.694013 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:04.693823    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph_odh-ai-gateway-infra(4df82c44-b352-4aad-9359-2cff6d73054f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" podUID="4df82c44-b352-4aad-9359-2cff6d73054f"
Oct 31 01:45:04.695159 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:04.695140    5718 scope.go:122] "RemoveContainer" containerID="795d0b6f37fe23331d7ccc9f4f8e43ad4c7b91059f93cebe5ecfea60280f82e5"
Oct 31 01:45:04.695422 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:04.695395    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv_odh-ai-gateway-infra(46ae1c4a-a518-4c08-9543-cb5ead10f9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0"
Oct 31 01:45:04.696885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:04.696866    5718 scope.go:122] "RemoveContainer" containerID="0de341bb05366ae5caab03265b15f9f40be887bf7b970ef7bf6972df454fd4bd"
Oct 31 01:45:04.697151 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:04.697125    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b_odh-ai-gateway-infra(65f80574-077d-485b-a6bb-78874aaee118)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" podUID="65f80574-077d-485b-a6bb-78874aaee118"
Oct 31 01:45:04.698380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:04.698363    5718 scope.go:122] "RemoveContainer" containerID="e68220675328de16c197e21595c535c385b803d80f137adafe07433b92289602"
Oct 31 01:45:04.698563 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:04.698546    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890185-hqwjp_odh-ai-gateway-infra(066229e9-7c64-4c3e-8e84-cae318086c20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" podUID="066229e9-7c64-4c3e-8e84-cae318086c20"
Oct 31 01:45:07.923831 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:07.923787    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:45:08.102727 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.102682    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"]
Oct 31 01:45:08.106602 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.106585    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.159397 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.159361    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8a3796-serving-cert\""
Oct 31 01:45:08.212338 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.212230    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"]
Oct 31 01:45:08.221527 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.221488    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.221763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.221567    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cln\" (UniqueName: \"kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.221763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.221645    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.221763 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.221727    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.322487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.322452    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.322736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.322510    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cln\" (UniqueName: \"kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.322736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.322534    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.322736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.322584    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.322926 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:08.322741    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-serving-cert: secret "maas-api-e2e-praxis-a-8a3796-serving-cert" not found
Oct 31 01:45:08.322926 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:08.322808    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls podName:f90bdfe8-cc91-4f35-973b-ca7f921e2f31 nodeName:}" failed. No retries permitted until 2026-10-31 01:45:08.82279064 +0000 UTC m=+2424.985934047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls") pod "maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" (UID: "f90bdfe8-cc91-4f35-973b-ca7f921e2f31") : secret "maas-api-e2e-praxis-a-8a3796-serving-cert" not found
Oct 31 01:45:08.323198 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.323175    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.325205 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.325182    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.381208 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.381173    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cln\" (UniqueName: \"kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.829295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.829255    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:08.831904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:08.831882    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:09.020077 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:09.020034    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:09.399736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:09.399704    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"]
Oct 31 01:45:09.401344 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:09.401317    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90bdfe8_cc91_4f35_973b_ca7f921e2f31.slice/crio-25a3ac778be3aec6f6ce4ca65f1101fa81b66a6a1a017b2757845eb8949873ea WatchSource:0}: Error finding container 25a3ac778be3aec6f6ce4ca65f1101fa81b66a6a1a017b2757845eb8949873ea: Status 404 returned error can't find the container with id 25a3ac778be3aec6f6ce4ca65f1101fa81b66a6a1a017b2757845eb8949873ea
Oct 31 01:45:09.402874 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:09.402856    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:45:09.734024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:09.733928    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" event={"ID":"f90bdfe8-cc91-4f35-973b-ca7f921e2f31","Type":"ContainerStarted","Data":"25a3ac778be3aec6f6ce4ca65f1101fa81b66a6a1a017b2757845eb8949873ea"}
Oct 31 01:45:10.654009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:10.653978    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:10.667058 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:10.667019    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:10.741596 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:10.741549    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" event={"ID":"f90bdfe8-cc91-4f35-973b-ca7f921e2f31","Type":"ContainerStarted","Data":"44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52"}
Oct 31 01:45:10.741806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:10.741786    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:10.754526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:10.754494    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:45:11.015980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:11.015927    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" podStartSLOduration=2.200797191 podStartE2EDuration="3.015909542s" totalImagesPullingTime="815.112351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:11.013982016 +0000 UTC m=+2427.177125425" watchObservedRunningTime="2026-10-31 01:45:11.015909542 +0000 UTC m=+2427.179052957"
Oct 31 01:45:14.960156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:14.960118    5718 scope.go:122] "RemoveContainer" containerID="f3afd7c9c71c08a2aab923e4c3961c01766ae595b05d528f8e93a67b9d21aaff"
Oct 31 01:45:14.960746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:14.960203    5718 scope.go:122] "RemoveContainer" containerID="0de341bb05366ae5caab03265b15f9f40be887bf7b970ef7bf6972df454fd4bd"
Oct 31 01:45:14.960746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:14.960368    5718 scope.go:122] "RemoveContainer" containerID="795d0b6f37fe23331d7ccc9f4f8e43ad4c7b91059f93cebe5ecfea60280f82e5"
Oct 31 01:45:15.773355 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:15.773314    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerStarted","Data":"dacd7c472cb9063684618cab29257da5c1a6fe5240f9d28a7ac43f2f973a5e64"}
Oct 31 01:45:15.775614 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:15.775589    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerStarted","Data":"16de8386d1ab7729eb0086804d4f600c8b688902bc857691570f82b806e9195d"}
Oct 31 01:45:15.777690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:15.777666    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerStarted","Data":"340f9d047d84204e6ca90d9f660539b99f7ace5f1b58ec4a74d3f960d7d5f652"}
Oct 31 01:45:16.752638 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.752599    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:45:16.786530 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.786496    5718 generic.go:356] "Generic (PLEG): container finished" podID="65f80574-077d-485b-a6bb-78874aaee118" containerID="16de8386d1ab7729eb0086804d4f600c8b688902bc857691570f82b806e9195d" exitCode=60
Oct 31 01:45:16.786722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.786565    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerDied","Data":"16de8386d1ab7729eb0086804d4f600c8b688902bc857691570f82b806e9195d"}
Oct 31 01:45:16.786722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.786624    5718 scope.go:122] "RemoveContainer" containerID="0de341bb05366ae5caab03265b15f9f40be887bf7b970ef7bf6972df454fd4bd"
Oct 31 01:45:16.787005 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.786984    5718 scope.go:122] "RemoveContainer" containerID="16de8386d1ab7729eb0086804d4f600c8b688902bc857691570f82b806e9195d"
Oct 31 01:45:16.787349 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:16.787314    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b_odh-ai-gateway-infra(65f80574-077d-485b-a6bb-78874aaee118)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" podUID="65f80574-077d-485b-a6bb-78874aaee118"
Oct 31 01:45:16.788752 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.788731    5718 generic.go:356] "Generic (PLEG): container finished" podID="4df82c44-b352-4aad-9359-2cff6d73054f" containerID="340f9d047d84204e6ca90d9f660539b99f7ace5f1b58ec4a74d3f960d7d5f652" exitCode=60
Oct 31 01:45:16.788838 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.788779    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerDied","Data":"340f9d047d84204e6ca90d9f660539b99f7ace5f1b58ec4a74d3f960d7d5f652"}
Oct 31 01:45:16.789138 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.789121    5718 scope.go:122] "RemoveContainer" containerID="340f9d047d84204e6ca90d9f660539b99f7ace5f1b58ec4a74d3f960d7d5f652"
Oct 31 01:45:16.789419 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:16.789394    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph_odh-ai-gateway-infra(4df82c44-b352-4aad-9359-2cff6d73054f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" podUID="4df82c44-b352-4aad-9359-2cff6d73054f"
Oct 31 01:45:16.791576 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.791556    5718 generic.go:356] "Generic (PLEG): container finished" podID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerID="dacd7c472cb9063684618cab29257da5c1a6fe5240f9d28a7ac43f2f973a5e64" exitCode=60
Oct 31 01:45:16.791747 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.791597    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerDied","Data":"dacd7c472cb9063684618cab29257da5c1a6fe5240f9d28a7ac43f2f973a5e64"}
Oct 31 01:45:16.791943 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.791930    5718 scope.go:122] "RemoveContainer" containerID="dacd7c472cb9063684618cab29257da5c1a6fe5240f9d28a7ac43f2f973a5e64"
Oct 31 01:45:16.792186 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:16.792168    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv_odh-ai-gateway-infra(46ae1c4a-a518-4c08-9543-cb5ead10f9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0"
Oct 31 01:45:16.799279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.799257    5718 scope.go:122] "RemoveContainer" containerID="f3afd7c9c71c08a2aab923e4c3961c01766ae595b05d528f8e93a67b9d21aaff"
Oct 31 01:45:16.811847 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.811828    5718 scope.go:122] "RemoveContainer" containerID="795d0b6f37fe23331d7ccc9f4f8e43ad4c7b91059f93cebe5ecfea60280f82e5"
Oct 31 01:45:16.913744 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.913715    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"]
Oct 31 01:45:16.960149 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:16.960123    5718 scope.go:122] "RemoveContainer" containerID="e68220675328de16c197e21595c535c385b803d80f137adafe07433b92289602"
Oct 31 01:45:17.018748 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:17.018646    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"]
Oct 31 01:45:17.129940 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:17.129904    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"]
Oct 31 01:45:17.802562 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:17.802524    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerStarted","Data":"7d558ed1618dee73778ffb97553acfe7ba27b0bc33505052f717fd6956399f48"}
Oct 31 01:45:17.962237 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:17.962212    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:18.007139 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.007116    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:18.040484 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.040455    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca\") pod \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") "
Oct 31 01:45:18.040662 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.040516    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca\") pod \"65f80574-077d-485b-a6bb-78874aaee118\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") "
Oct 31 01:45:18.040662 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.040542    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz\" (UniqueName: \"kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz\") pod \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\" (UID: \"46ae1c4a-a518-4c08-9543-cb5ead10f9f0\") "
Oct 31 01:45:18.040662 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.040608    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g\" (UniqueName: \"kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g\") pod \"65f80574-077d-485b-a6bb-78874aaee118\" (UID: \"65f80574-077d-485b-a6bb-78874aaee118\") "
Oct 31 01:45:18.040859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.040833    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca" pod "46ae1c4a-a518-4c08-9543-cb5ead10f9f0" (UID: "46ae1c4a-a518-4c08-9543-cb5ead10f9f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:18.040918 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.040879    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca" pod "65f80574-077d-485b-a6bb-78874aaee118" (UID: "65f80574-077d-485b-a6bb-78874aaee118"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:18.041475 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.041439    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.041475 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.041463    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f80574-077d-485b-a6bb-78874aaee118-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.042937 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.042914    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz" pod "46ae1c4a-a518-4c08-9543-cb5ead10f9f0" (UID: "46ae1c4a-a518-4c08-9543-cb5ead10f9f0"). InnerVolumeSpecName "kube-api-access-sq8wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:18.043050 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.042982    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g" pod "65f80574-077d-485b-a6bb-78874aaee118" (UID: "65f80574-077d-485b-a6bb-78874aaee118"). InnerVolumeSpecName "kube-api-access-9l57g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:18.059399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.059377    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:18.142548 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.142505    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca\") pod \"4df82c44-b352-4aad-9359-2cff6d73054f\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") "
Oct 31 01:45:18.142777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.142721    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz\" (UniqueName: \"kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz\") pod \"4df82c44-b352-4aad-9359-2cff6d73054f\" (UID: \"4df82c44-b352-4aad-9359-2cff6d73054f\") "
Oct 31 01:45:18.142943 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.142912    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca" pod "4df82c44-b352-4aad-9359-2cff6d73054f" (UID: "4df82c44-b352-4aad-9359-2cff6d73054f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:18.143165 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.143138    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l57g\" (UniqueName: \"kubernetes.io/projected/65f80574-077d-485b-a6bb-78874aaee118-kube-api-access-9l57g\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.143244 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.143163    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8wz\" (UniqueName: \"kubernetes.io/projected/46ae1c4a-a518-4c08-9543-cb5ead10f9f0-kube-api-access-sq8wz\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.143244 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.143181    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df82c44-b352-4aad-9359-2cff6d73054f-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.145297 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.145268    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz" pod "4df82c44-b352-4aad-9359-2cff6d73054f" (UID: "4df82c44-b352-4aad-9359-2cff6d73054f"). InnerVolumeSpecName "kube-api-access-6z5dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:18.244261 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.244228    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5dz\" (UniqueName: \"kubernetes.io/projected/4df82c44-b352-4aad-9359-2cff6d73054f-kube-api-access-6z5dz\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.813726 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.813686    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"
Oct 31 01:45:18.813726 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.813689    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv" event={"ID":"46ae1c4a-a518-4c08-9543-cb5ead10f9f0","Type":"ContainerDied","Data":"b69dd513a02cc3210030230140effeae7d5329f46f690458de245132bcc7d4b6"}
Oct 31 01:45:18.814294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.813799    5718 scope.go:122] "RemoveContainer" containerID="dacd7c472cb9063684618cab29257da5c1a6fe5240f9d28a7ac43f2f973a5e64"
Oct 31 01:45:18.820275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.820251    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"
Oct 31 01:45:18.820397 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.820251    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b" event={"ID":"65f80574-077d-485b-a6bb-78874aaee118","Type":"ContainerDied","Data":"db74ccec60c2abbf9fc9e5915702d552d136eb58ac47267ca3f87672df565666"}
Oct 31 01:45:18.827127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.823621    5718 generic.go:356] "Generic (PLEG): container finished" podID="066229e9-7c64-4c3e-8e84-cae318086c20" containerID="7d558ed1618dee73778ffb97553acfe7ba27b0bc33505052f717fd6956399f48" exitCode=60
Oct 31 01:45:18.827127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.823762    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerDied","Data":"7d558ed1618dee73778ffb97553acfe7ba27b0bc33505052f717fd6956399f48"}
Oct 31 01:45:18.827127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.824134    5718 scope.go:122] "RemoveContainer" containerID="7d558ed1618dee73778ffb97553acfe7ba27b0bc33505052f717fd6956399f48"
Oct 31 01:45:18.827127 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:18.824434    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890185-hqwjp_odh-ai-gateway-infra(066229e9-7c64-4c3e-8e84-cae318086c20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" podUID="066229e9-7c64-4c3e-8e84-cae318086c20"
Oct 31 01:45:18.827127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.826506    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph" event={"ID":"4df82c44-b352-4aad-9359-2cff6d73054f","Type":"ContainerDied","Data":"529a37bffba4d2ba358afaaa743f1583455bd5e1bd0dc7003064d225e565582b"}
Oct 31 01:45:18.827127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.826617    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"
Oct 31 01:45:18.833059 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.833039    5718 scope.go:122] "RemoveContainer" containerID="16de8386d1ab7729eb0086804d4f600c8b688902bc857691570f82b806e9195d"
Oct 31 01:45:18.842535 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.842517    5718 scope.go:122] "RemoveContainer" containerID="e68220675328de16c197e21595c535c385b803d80f137adafe07433b92289602"
Oct 31 01:45:18.855125 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:18.855107    5718 scope.go:122] "RemoveContainer" containerID="340f9d047d84204e6ca90d9f660539b99f7ace5f1b58ec4a74d3f960d7d5f652"
Oct 31 01:45:19.046537 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.046507    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"]
Oct 31 01:45:19.108848 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.108761    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"]
Oct 31 01:45:19.112617 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.112581    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890185-6rxph"]
Oct 31 01:45:19.228213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.228172    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"]
Oct 31 01:45:19.230344 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.230318    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890185-5qb5b"]
Oct 31 01:45:19.340059 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.340022    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"]
Oct 31 01:45:19.345301 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.345265    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890185-j96dv"]
Oct 31 01:45:19.607951 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.607915    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"]
Oct 31 01:45:19.608395 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608382    5718 cpu_manager.go:414] "containerMap: removing container" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608396    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608406    5718 cpu_manager.go:414] "containerMap: removing container" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:45:19.608441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608411    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:45:19.608441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608426    5718 cpu_manager.go:414] "containerMap: removing container" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:45:19.608441 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608431    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608450    5718 cpu_manager.go:414] "containerMap: removing container" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608455    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608464    5718 cpu_manager.go:414] "containerMap: removing container" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608468    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608475    5718 cpu_manager.go:414] "containerMap: removing container" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608480    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608488    5718 cpu_manager.go:414] "containerMap: removing container" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608493    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608500    5718 cpu_manager.go:414] "containerMap: removing container" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608505    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608566    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608574    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608583    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608588    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608595    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608600    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:45:19.608599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.608605    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f80574-077d-485b-a6bb-78874aaee118" containerName="cleanup"
Oct 31 01:45:19.612867 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.612848    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.659690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.659456    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.659690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.659570    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.659690 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.659632    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29mpn\" (UniqueName: \"kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.659979 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.659771    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.664606 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.664572    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"]
Oct 31 01:45:19.665525 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.665494    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c116f3-serving-cert\""
Oct 31 01:45:19.760803 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.760770    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.760980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.760834    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mpn\" (UniqueName: \"kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.760980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.760896    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.761121 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.761036    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.761220 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:19.761197    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-serving-cert: secret "maas-api-e2e-praxis-a-c116f3-serving-cert" not found
Oct 31 01:45:19.761285 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:19.761258    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls podName:bd418b2e-650b-438a-899a-9dbe22e92099 nodeName:}" failed. No retries permitted until 2026-10-31 01:45:20.261243024 +0000 UTC m=+2436.424386418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls") pod "maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" (UID: "bd418b2e-650b-438a-899a-9dbe22e92099") : secret "maas-api-e2e-praxis-a-c116f3-serving-cert" not found
Oct 31 01:45:19.761555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.761534    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.763563 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.763540    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.819057 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.819028    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mpn\" (UniqueName: \"kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:19.970327 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.970295    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" path="/var/lib/kubelet/pods/46ae1c4a-a518-4c08-9543-cb5ead10f9f0/volumes"
Oct 31 01:45:19.970739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.970722    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" path="/var/lib/kubelet/pods/4df82c44-b352-4aad-9359-2cff6d73054f/volumes"
Oct 31 01:45:19.971087 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:19.971073    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f80574-077d-485b-a6bb-78874aaee118" path="/var/lib/kubelet/pods/65f80574-077d-485b-a6bb-78874aaee118/volumes"
Oct 31 01:45:20.007859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.007833    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:20.064692 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.064643    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca\") pod \"066229e9-7c64-4c3e-8e84-cae318086c20\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") "
Oct 31 01:45:20.064853 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.064810    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5\" (UniqueName: \"kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5\") pod \"066229e9-7c64-4c3e-8e84-cae318086c20\" (UID: \"066229e9-7c64-4c3e-8e84-cae318086c20\") "
Oct 31 01:45:20.065013 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.064993    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca" pod "066229e9-7c64-4c3e-8e84-cae318086c20" (UID: "066229e9-7c64-4c3e-8e84-cae318086c20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:20.065217 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.065204    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/066229e9-7c64-4c3e-8e84-cae318086c20-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:20.067091 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.067035    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5" pod "066229e9-7c64-4c3e-8e84-cae318086c20" (UID: "066229e9-7c64-4c3e-8e84-cae318086c20"). InnerVolumeSpecName "kube-api-access-77lg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:20.167008 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.166929    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77lg5\" (UniqueName: \"kubernetes.io/projected/066229e9-7c64-4c3e-8e84-cae318086c20-kube-api-access-77lg5\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:20.268475 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.268437    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:20.271180 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.271157    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:20.526234 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.526193    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:20.844057 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.843969    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp" event={"ID":"066229e9-7c64-4c3e-8e84-cae318086c20","Type":"ContainerDied","Data":"9e50d269bb96cf12e5a7c8731ce1a229afdbdd98292bed031f113d38d699dfd3"}
Oct 31 01:45:20.844057 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.844030    5718 scope.go:122] "RemoveContainer" containerID="7d558ed1618dee73778ffb97553acfe7ba27b0bc33505052f717fd6956399f48"
Oct 31 01:45:20.844520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.843981    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"
Oct 31 01:45:20.904844 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:45:20.904812    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd418b2e_650b_438a_899a_9dbe22e92099.slice/crio-e1cae93c48d4f844608ab4955089dee68d3ee4309998a3e6bb34256c305b2287 WatchSource:0}: Error finding container e1cae93c48d4f844608ab4955089dee68d3ee4309998a3e6bb34256c305b2287: Status 404 returned error can't find the container with id e1cae93c48d4f844608ab4955089dee68d3ee4309998a3e6bb34256c305b2287
Oct 31 01:45:20.950889 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:20.950859    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"]
Oct 31 01:45:21.012231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.012196    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"]
Oct 31 01:45:21.016277 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.016252    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890185-hqwjp"]
Oct 31 01:45:21.853809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.853773    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" event={"ID":"bd418b2e-650b-438a-899a-9dbe22e92099","Type":"ContainerStarted","Data":"bad1ea84f3149010b5e33aae1fca1d998a4942b572e54112e9f33462aa9f662a"}
Oct 31 01:45:21.854174 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.853927    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:21.854174 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.853968    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" event={"ID":"bd418b2e-650b-438a-899a-9dbe22e92099","Type":"ContainerStarted","Data":"e1cae93c48d4f844608ab4955089dee68d3ee4309998a3e6bb34256c305b2287"}
Oct 31 01:45:21.967294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.967203    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" path="/var/lib/kubelet/pods/066229e9-7c64-4c3e-8e84-cae318086c20/volumes"
Oct 31 01:45:21.997896 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:21.997515    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" podStartSLOduration=2.207581671 podStartE2EDuration="2.997500012s" totalImagesPullingTime="789.918341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:21.997233794 +0000 UTC m=+2438.160377210" watchObservedRunningTime="2026-10-31 01:45:21.997500012 +0000 UTC m=+2438.160643428"
Oct 31 01:45:27.865054 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:45:27.864975    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:45:37.923948 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:45:37.923908    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:46:02.545787 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.545750    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"]
Oct 31 01:46:02.546315 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546301    5718 cpu_manager.go:414] "containerMap: removing container" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:46:02.546359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546317    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:46:02.546359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546335    5718 cpu_manager.go:414] "containerMap: removing container" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546341    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546360    5718 cpu_manager.go:414] "containerMap: removing container" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546369    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546382    5718 cpu_manager.go:414] "containerMap: removing container" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546387    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546482    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546490    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df82c44-b352-4aad-9359-2cff6d73054f" containerName="cleanup"
Oct 31 01:46:02.546569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546496    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ae1c4a-a518-4c08-9543-cb5ead10f9f0" containerName="cleanup"
Oct 31 01:46:02.546569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546503    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.546569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.546510    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="066229e9-7c64-4c3e-8e84-cae318086c20" containerName="cleanup"
Oct 31 01:46:02.551840 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.551819    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.609980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.609945    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c116f3-openshift-default-dockercfg-289mh\""
Oct 31 01:46:02.673772 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.673736    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.673801    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.673895    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.674025    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674163 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.674103    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjh9\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.674203    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674266 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.674252    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674371 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.674314    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.674462 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.674443    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.756003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.755914    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"]
Oct 31 01:46:02.775734 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.775700    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.775892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.775753    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.775892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.775855    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.775982 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.775964    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776055 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776033    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776196 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776082    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776196 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776114    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776196 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776187    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776213    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776116    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776261    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjh9\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776359 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776301    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776601 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776575    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.776693 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.776624    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.778809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.778780    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.784780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.784758    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.833565 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.833493    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.833717 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.833608    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjh9\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:02.864854 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:02.864829    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:03.294004 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:03.293961    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"]
Oct 31 01:46:03.295975 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:46:03.295943    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdda880_ef9c_4c59_aa6f_52640e0652ed.slice/crio-73260b2f7864969ddf2537801ec259e5c4576f65c0dcd729e9295afb3137a9cd WatchSource:0}: Error finding container 73260b2f7864969ddf2537801ec259e5c4576f65c0dcd729e9295afb3137a9cd: Status 404 returned error can't find the container with id 73260b2f7864969ddf2537801ec259e5c4576f65c0dcd729e9295afb3137a9cd
Oct 31 01:46:03.298538 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:03.298507    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:46:03.298620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:03.298567    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:46:03.298620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:03.298597    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:46:04.099871 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.099823    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" event={"ID":"0cdda880-ef9c-4c59-aa6f-52640e0652ed","Type":"ContainerStarted","Data":"749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b"}
Oct 31 01:46:04.100324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.099899    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" event={"ID":"0cdda880-ef9c-4c59-aa6f-52640e0652ed","Type":"ContainerStarted","Data":"73260b2f7864969ddf2537801ec259e5c4576f65c0dcd729e9295afb3137a9cd"}
Oct 31 01:46:04.224263 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.224214    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podStartSLOduration=2.2241992 podStartE2EDuration="2.2241992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:04.220955235 +0000 UTC m=+2480.384098648" watchObservedRunningTime="2026-10-31 01:46:04.2241992 +0000 UTC m=+2480.387342614"
Oct 31 01:46:04.611156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.611119    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"]
Oct 31 01:46:04.615113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.615088    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.668093 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.668052    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-9d7b2c-serving-cert\""
Oct 31 01:46:04.690517 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.690484    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"]
Oct 31 01:46:04.702463 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.702428    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.702639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.702487    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.702639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.702591    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.702639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.702635    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ng7\" (UniqueName: \"kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.803317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.803283    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.803516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.803336    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ng7\" (UniqueName: \"kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.803516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.803385    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.803516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.803414    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.803720 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:46:04.803568    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-serving-cert: secret "maas-api-e2e-praxis-a-9d7b2c-serving-cert" not found
Oct 31 01:46:04.803720 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:46:04.803710    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls podName:cf7fa906-e80d-49f4-972f-e635136cccb3 nodeName:}" failed. No retries permitted until 2026-10-31 01:46:05.303639649 +0000 UTC m=+2481.466783057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls") pod "maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" (UID: "cf7fa906-e80d-49f4-972f-e635136cccb3") : secret "maas-api-e2e-praxis-a-9d7b2c-serving-cert" not found
Oct 31 01:46:04.804053 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.804027    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.806222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.806200    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:04.865948 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.865856    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:04.867929 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.867890    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:46:04.868129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.867951    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:46:04.869982 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:04.869946    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ng7\" (UniqueName: \"kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:05.309046 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:05.308995    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:05.311568 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:05.311543    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:05.529289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:05.529257    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:05.870510 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:05.870396    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:46:05.870510 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:05.870493    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:46:05.939528 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:05.939499    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"]
Oct 31 01:46:06.111728 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:06.111692    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" event={"ID":"cf7fa906-e80d-49f4-972f-e635136cccb3","Type":"ContainerStarted","Data":"5fcf7b0df24aa661f412974b3dceff30864205d1c2e2e02283e2de543407487f"}
Oct 31 01:46:06.868677 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:06.868054    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:46:06.868677 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:06.868120    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:46:07.119778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:07.119683    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" event={"ID":"cf7fa906-e80d-49f4-972f-e635136cccb3","Type":"ContainerStarted","Data":"d7e4cd50d6240d0453d31bf0586bdf784fe2142082650130260e5c5e570c2cf0"}
Oct 31 01:46:07.119778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:07.119753    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:07.238276 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:07.238222    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" podStartSLOduration=2.402655006 podStartE2EDuration="3.238208978s" totalImagesPullingTime="835.553972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:07.235776353 +0000 UTC m=+2483.398919769" watchObservedRunningTime="2026-10-31 01:46:07.238208978 +0000 UTC m=+2483.401352430"
Oct 31 01:46:07.865312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:07.865257    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:46:07.865770 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:07.865724    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:46:07.924680 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:46:07.924622    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:46:08.866226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:08.866179    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:46:08.866434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:08.866248    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 01:46:09.870544 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:09.870508    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:09.870973 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:09.870860    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:09.871524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:09.871506    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:46:13.131746 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:13.131712    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:46:32.779816 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.779784    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"]
Oct 31 01:46:32.783798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.783780    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860205    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860270    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860395    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860434    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860580    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860643 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860626    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860739    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860806    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57zf\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.860858 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.860834    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.907974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.902162    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"]
Oct 31 01:46:32.961859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.961822    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57zf\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.961874    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.961946    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.961968    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962215 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962130    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962215 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962174    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962220    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962257    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962313    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962367    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962394    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962505 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962474    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962621 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962562    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.962968 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.962947    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.964618 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.964596    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:32.964778 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:32.964740    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:33.020258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.020227    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:33.020456 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.020425    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57zf\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:33.098432 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.098338    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:33.548810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.548783    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"]
Oct 31 01:46:33.551289 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:46:33.551260    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0884023c_eaf6_413e_aded_abdfb59c064f.slice/crio-5b3887549c0ee93551b8038cf6b31b7f5ff6a8978bb2e212c21c2389d37f6e2b WatchSource:0}: Error finding container 5b3887549c0ee93551b8038cf6b31b7f5ff6a8978bb2e212c21c2389d37f6e2b: Status 404 returned error can't find the container with id 5b3887549c0ee93551b8038cf6b31b7f5ff6a8978bb2e212c21c2389d37f6e2b
Oct 31 01:46:33.553914 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.553881    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:46:33.554005 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.553946    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:46:33.554005 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:33.553977    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:46:34.282143 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:34.282107    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" event={"ID":"0884023c-eaf6-413e-aded-abdfb59c064f","Type":"ContainerStarted","Data":"a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c"}
Oct 31 01:46:34.282540 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:34.282169    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" event={"ID":"0884023c-eaf6-413e-aded-abdfb59c064f","Type":"ContainerStarted","Data":"5b3887549c0ee93551b8038cf6b31b7f5ff6a8978bb2e212c21c2389d37f6e2b"}
Oct 31 01:46:34.404486 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:34.404433    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podStartSLOduration=2.404419402 podStartE2EDuration="2.404419402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:34.401978532 +0000 UTC m=+2510.565121947" watchObservedRunningTime="2026-10-31 01:46:34.404419402 +0000 UTC m=+2510.567562816"
Oct 31 01:46:35.099604 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:35.099564    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:35.101451 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:35.101424    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:46:35.101585 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:35.101478    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 01:46:36.099499 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:36.099461    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:46:36.099972 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:36.099533    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 01:46:37.110010 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:37.100850    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:46:37.110010 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:37.100922    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 01:46:37.928115 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:46:37.928076    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:46:38.099358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:38.099312    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:46:38.099533 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:38.099398    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 01:46:39.099340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:39.099304    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:46:39.099775 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:39.099360    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 01:46:40.104361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:40.104322    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:40.104976 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:40.104954    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:46:40.105942 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:46:40.105920    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:47:07.925435 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:47:07.925341    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:47:37.923743 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:47:37.923701    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:47:50.299539 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.299500    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"]
Oct 31 01:47:50.306110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.306076    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.361597 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.361565    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"]
Oct 31 01:47:50.409245 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.409209    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e59f12-serving-cert\""
Oct 31 01:47:50.409799 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.409769    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.409923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.409907    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.409961 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.409949    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnph\" (UniqueName: \"kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.410064 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.410050    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.511445 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.511407    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.511699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.511508    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.511699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.511553    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnph\" (UniqueName: \"kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.511699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.511615    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.511699 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:47:50.511644    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-serving-cert: secret "maas-api-e2e-praxis-b-e59f12-serving-cert" not found
Oct 31 01:47:50.511969 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:47:50.511748    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls podName:6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541 nodeName:}" failed. No retries permitted until 2026-10-31 01:47:51.011727335 +0000 UTC m=+2587.174870732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls") pod "maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" (UID: "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541") : secret "maas-api-e2e-praxis-b-e59f12-serving-cert" not found
Oct 31 01:47:50.512378 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.512350    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.514949 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.514921    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:50.569980 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:50.569908    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnph\" (UniqueName: \"kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:51.017598 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:51.017556    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:51.020438 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:51.020409    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:51.222830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:51.222786    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:51.602329 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:51.602291    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"]
Oct 31 01:47:51.603537 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:47:51.603506    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfcaa6f_3cc3_4b0f_b3fe_a47ba0eeb541.slice/crio-3ab5e3db7cee27dd452d00b89336690434518c6250a9d8e00e5d32cc55b1732d WatchSource:0}: Error finding container 3ab5e3db7cee27dd452d00b89336690434518c6250a9d8e00e5d32cc55b1732d: Status 404 returned error can't find the container with id 3ab5e3db7cee27dd452d00b89336690434518c6250a9d8e00e5d32cc55b1732d
Oct 31 01:47:51.772919 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:51.772879    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" event={"ID":"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541","Type":"ContainerStarted","Data":"3ab5e3db7cee27dd452d00b89336690434518c6250a9d8e00e5d32cc55b1732d"}
Oct 31 01:47:52.780689 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:52.780607    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" event={"ID":"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541","Type":"ContainerStarted","Data":"bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5"}
Oct 31 01:47:52.781193 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:52.780706    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:47:52.898450 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:52.898397    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" podStartSLOduration=2.022974414 podStartE2EDuration="2.898382346s" totalImagesPullingTime="875.407932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:52.896980302 +0000 UTC m=+2589.060123709" watchObservedRunningTime="2026-10-31 01:47:52.898382346 +0000 UTC m=+2589.061525757"
Oct 31 01:47:58.791941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:47:58.791907    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:48:05.029188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.029149    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"]
Oct 31 01:48:05.033418 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.033388    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.068993 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.068957    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.069401 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.069377    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.069570 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.069543    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkv4p\" (UniqueName: \"kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.070268 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.070243    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.087897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.087864    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-99145d-serving-cert\""
Oct 31 01:48:05.140738 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.140708    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"]
Oct 31 01:48:05.171486 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.171446    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.171704 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.171535    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.171704 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.171646    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.171831 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.171738    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkv4p\" (UniqueName: \"kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.171887 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:48:05.171828    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-serving-cert: secret "maas-api-e2e-praxis-b-99145d-serving-cert" not found
Oct 31 01:48:05.171943 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:48:05.171902    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls podName:3b40ca9c-6716-4f94-9719-5313ef504b5a nodeName:}" failed. No retries permitted until 2026-10-31 01:48:05.671878519 +0000 UTC m=+2601.835021924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls") pod "maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" (UID: "3b40ca9c-6716-4f94-9719-5313ef504b5a") : secret "maas-api-e2e-praxis-b-99145d-serving-cert" not found
Oct 31 01:48:05.172222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.172198    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.174411 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.174387    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.230044 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.230013    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkv4p\" (UniqueName: \"kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.676929 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.676895    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.679548 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.679524    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:05.947768 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:05.947636    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:06.331111 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:06.331077    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"]
Oct 31 01:48:06.332008 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:48:06.331978    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b40ca9c_6716_4f94_9719_5313ef504b5a.slice/crio-d95e43219730af64fd438f1ebe82a25744e5e491cd84607aad47bdf9db340df0 WatchSource:0}: Error finding container d95e43219730af64fd438f1ebe82a25744e5e491cd84607aad47bdf9db340df0: Status 404 returned error can't find the container with id d95e43219730af64fd438f1ebe82a25744e5e491cd84607aad47bdf9db340df0
Oct 31 01:48:06.862911 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:06.862868    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" event={"ID":"3b40ca9c-6716-4f94-9719-5313ef504b5a","Type":"ContainerStarted","Data":"d95e43219730af64fd438f1ebe82a25744e5e491cd84607aad47bdf9db340df0"}
Oct 31 01:48:07.871324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:07.871290    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" event={"ID":"3b40ca9c-6716-4f94-9719-5313ef504b5a","Type":"ContainerStarted","Data":"d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378"}
Oct 31 01:48:07.871860 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:07.871495    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:07.924762 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:48:07.924716    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:48:08.008802 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:08.008750    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" podStartSLOduration=3.132959682 podStartE2EDuration="4.008733168s" totalImagesPullingTime="875.773486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:08.005572783 +0000 UTC m=+2604.168716198" watchObservedRunningTime="2026-10-31 01:48:08.008733168 +0000 UTC m=+2604.171876627"
Oct 31 01:48:13.883293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:13.883261    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:48:37.924533 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:48:37.924487    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:48:45.204915 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.204866    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"]
Oct 31 01:48:45.208900 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.208873    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.263263 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.263230    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-cd0c98-serving-cert\""
Oct 31 01:48:45.266969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.266938    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"]
Oct 31 01:48:45.355855 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.355819    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.356037 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.355864    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.356118 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.356089    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcq6j\" (UniqueName: \"kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.356173 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.356157    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.457623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.457515    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.457623 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.457568    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.457904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.457739    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcq6j\" (UniqueName: \"kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.457904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.457785    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.458027 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:48:45.457906    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-serving-cert: secret "maas-api-e2e-praxis-b-cd0c98-serving-cert" not found
Oct 31 01:48:45.458027 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:48:45.457982    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls podName:2a1f06cc-9f39-4004-b87b-2859da22601d nodeName:}" failed. No retries permitted until 2026-10-31 01:48:45.957959678 +0000 UTC m=+2642.121103088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls") pod "maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" (UID: "2a1f06cc-9f39-4004-b87b-2859da22601d") : secret "maas-api-e2e-praxis-b-cd0c98-serving-cert" not found
Oct 31 01:48:45.458584 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.458554    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.461852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.461826    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.516524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.516483    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcq6j\" (UniqueName: \"kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.962575 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.962542    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:45.965218 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:45.965189    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:46.123461 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:46.123426    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:46.506153 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:48:46.506120    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1f06cc_9f39_4004_b87b_2859da22601d.slice/crio-2b9283475391d5123d0f14ee633f6eb398997654933f9e9121e8e411178c9085 WatchSource:0}: Error finding container 2b9283475391d5123d0f14ee633f6eb398997654933f9e9121e8e411178c9085: Status 404 returned error can't find the container with id 2b9283475391d5123d0f14ee633f6eb398997654933f9e9121e8e411178c9085
Oct 31 01:48:46.508497 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:46.508455    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"]
Oct 31 01:48:47.106128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:47.106091    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" event={"ID":"2a1f06cc-9f39-4004-b87b-2859da22601d","Type":"ContainerStarted","Data":"2b9283475391d5123d0f14ee633f6eb398997654933f9e9121e8e411178c9085"}
Oct 31 01:48:48.114509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:48.114458    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" event={"ID":"2a1f06cc-9f39-4004-b87b-2859da22601d","Type":"ContainerStarted","Data":"a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c"}
Oct 31 01:48:48.114944 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:48.114702    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:48:48.245128 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:48.245068    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" podStartSLOduration=2.435031637 podStartE2EDuration="3.245049026s" totalImagesPullingTime="810.017389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:48.242022008 +0000 UTC m=+2644.405165425" watchObservedRunningTime="2026-10-31 01:48:48.245049026 +0000 UTC m=+2644.408192453"
Oct 31 01:48:54.126304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:48:54.126270    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:49:07.924432 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:49:07.924390    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:49:18.402492 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.402429    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"]
Oct 31 01:49:18.406469 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.406453    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446053 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446017    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446057    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446078    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446154    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stkf\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446235 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446201    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446435 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446282    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446435 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446324    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446436    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.446520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.446468    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.459580 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.459544    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cd0c98-openshift-default-dockercfg-xf86x\""
Oct 31 01:49:18.547277 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547240    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547287    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547311    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547346    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stkf\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547390    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547418    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547481 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547472    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547584    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547636    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.547937 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.547846    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.548066 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.548018    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.548192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.548070    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.548192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.548099    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.548192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.548132    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.550065 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.550045    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.550153 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.550119    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.609110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.608573    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stkf\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.609110 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.609095    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"]
Oct 31 01:49:18.610910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.610886    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:18.720719 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:18.720628    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:19.151158 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.151130    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"]
Oct 31 01:49:19.152162 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:49:19.152137    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30d5d30_3032_4f23_ae72_8993c39980de.slice/crio-a64da2613949d991209a6f7336f4d648f89b9102380fcaaf1f5a913b4ee15efa WatchSource:0}: Error finding container a64da2613949d991209a6f7336f4d648f89b9102380fcaaf1f5a913b4ee15efa: Status 404 returned error can't find the container with id a64da2613949d991209a6f7336f4d648f89b9102380fcaaf1f5a913b4ee15efa
Oct 31 01:49:19.154769 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.154735    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:49:19.154842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.154802    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:49:19.154842 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.154829    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:49:19.296293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.296256    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" event={"ID":"f30d5d30-3032-4f23-ae72-8993c39980de","Type":"ContainerStarted","Data":"3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f"}
Oct 31 01:49:19.296512 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.296320    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" event={"ID":"f30d5d30-3032-4f23-ae72-8993c39980de","Type":"ContainerStarted","Data":"a64da2613949d991209a6f7336f4d648f89b9102380fcaaf1f5a913b4ee15efa"}
Oct 31 01:49:19.418756 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.418604    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podStartSLOduration=1.41858455 podStartE2EDuration="1.41858455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:19.418406678 +0000 UTC m=+2675.581550094" watchObservedRunningTime="2026-10-31 01:49:19.41858455 +0000 UTC m=+2675.581727965"
Oct 31 01:49:19.721496 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:19.721403    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:20.723092 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:20.723052    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:49:20.723609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:20.723112    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 01:49:21.721165 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:21.721124    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:49:21.721380 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:21.721193    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 01:49:22.721155 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:22.721118    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:49:22.721751 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:22.721714    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 01:49:23.721010 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:23.720971    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:49:23.721230 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:23.721033    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 01:49:24.721010 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:24.720967    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:49:24.721221 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:24.721027    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 01:49:25.726965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:25.726932    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:25.727387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:25.727326    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:25.728051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:49:25.728031    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:49:37.923773 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:49:37.923721    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:50:03.761802 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.761724    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"]
Oct 31 01:50:03.765589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.765563    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920493 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920460    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920689 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920582    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklpw\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920689 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920680    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920789 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920725    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920840 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920826    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920886 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920875    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920946 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920931    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.920999 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.920979    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.921049 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.921036    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:03.969526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:03.969495    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"]
Oct 31 01:50:04.022504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022408    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklpw\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.022504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022492    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.022780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022521    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.022780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022575    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.022780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022633    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.022780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022749    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.022780 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022780    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022814    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.022891    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.023189    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023291 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.023267    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023325 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.023304    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023358 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.023330    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.023799 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.023774    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.025173 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.025149    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.025387 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.025369    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.079869 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.079842    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.080020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.079961    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklpw\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.379774 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.379680    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:04.810249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.810220    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"]
Oct 31 01:50:04.810643 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:50:04.810623    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4feddf28_bfd9_4eba_8b81_6a56e8bbdc0b.slice/crio-60230415af972480bbf5606d6be84aae3114bfec7d429be7e58c872e2242d1eb WatchSource:0}: Error finding container 60230415af972480bbf5606d6be84aae3114bfec7d429be7e58c872e2242d1eb: Status 404 returned error can't find the container with id 60230415af972480bbf5606d6be84aae3114bfec7d429be7e58c872e2242d1eb
Oct 31 01:50:04.813271 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.813240    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:50:04.813349 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.813307    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:50:04.813349 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:04.813338    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:50:05.575992 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:05.575947    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" event={"ID":"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b","Type":"ContainerStarted","Data":"1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1"}
Oct 31 01:50:05.576191 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:05.576024    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" event={"ID":"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b","Type":"ContainerStarted","Data":"60230415af972480bbf5606d6be84aae3114bfec7d429be7e58c872e2242d1eb"}
Oct 31 01:50:05.699157 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:05.699104    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podStartSLOduration=2.699090421 podStartE2EDuration="2.699090421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:05.694987324 +0000 UTC m=+2721.858130763" watchObservedRunningTime="2026-10-31 01:50:05.699090421 +0000 UTC m=+2721.862233836"
Oct 31 01:50:06.380285 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:06.380247    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:06.382175 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:06.382147    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:50:06.382310 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:06.382209    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 01:50:07.380595 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:07.380549    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:50:07.381018 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:07.380614    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 01:50:07.924310 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:50:07.924270    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:50:08.381036 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:08.381003    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:50:08.381440 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:08.381059    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 01:50:09.380231 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:09.380195    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:50:09.380420 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:09.380265    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 01:50:10.380906 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:10.380860    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 01:50:10.381337 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:10.380934    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 01:50:11.386413 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:11.386384    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:11.386839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:11.386783    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:11.387617 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:11.387596    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:50:25.240990 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.240958    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"]
Oct 31 01:50:25.241400 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.241138    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" podUID="f90bdfe8-cc91-4f35-973b-ca7f921e2f31" containerName="maas-api" containerID="cri-o://44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52" gracePeriod=30
Oct 31 01:50:25.544392 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.544356    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:50:25.639160 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.638626    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs\") pod \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") "
Oct 31 01:50:25.640781 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.639458    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\") pod \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") "
Oct 31 01:50:25.641064 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.641044    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln\" (UniqueName: \"kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln\") pod \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") "
Oct 31 01:50:25.642844 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.642764    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca\") pod \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\" (UID: \"f90bdfe8-cc91-4f35-973b-ca7f921e2f31\") "
Oct 31 01:50:25.643917 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.643891    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca" pod "f90bdfe8-cc91-4f35-973b-ca7f921e2f31" (UID: "f90bdfe8-cc91-4f35-973b-ca7f921e2f31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:25.664721 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.661361    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs" pod "f90bdfe8-cc91-4f35-973b-ca7f921e2f31" (UID: "f90bdfe8-cc91-4f35-973b-ca7f921e2f31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:25.665706 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.665365    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln" pod "f90bdfe8-cc91-4f35-973b-ca7f921e2f31" (UID: "f90bdfe8-cc91-4f35-973b-ca7f921e2f31"). InnerVolumeSpecName "kube-api-access-c8cln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:25.665706 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.665444    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls" pod "f90bdfe8-cc91-4f35-973b-ca7f921e2f31" (UID: "f90bdfe8-cc91-4f35-973b-ca7f921e2f31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:25.698939 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.698901    5718 generic.go:356] "Generic (PLEG): container finished" podID="f90bdfe8-cc91-4f35-973b-ca7f921e2f31" containerID="44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52" exitCode=0
Oct 31 01:50:25.699102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.698975    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" event={"ID":"f90bdfe8-cc91-4f35-973b-ca7f921e2f31","Type":"ContainerDied","Data":"44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52"}
Oct 31 01:50:25.699102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.699007    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"
Oct 31 01:50:25.699102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.699027    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp" event={"ID":"f90bdfe8-cc91-4f35-973b-ca7f921e2f31","Type":"ContainerDied","Data":"25a3ac778be3aec6f6ce4ca65f1101fa81b66a6a1a017b2757845eb8949873ea"}
Oct 31 01:50:25.699102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.699057    5718 scope.go:122] "RemoveContainer" containerID="44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52"
Oct 31 01:50:25.714373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.714349    5718 scope.go:122] "RemoveContainer" containerID="44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52"
Oct 31 01:50:25.714753 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:50:25.714732    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52\": container with ID starting with 44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52 not found: ID does not exist" containerID="44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52"
Oct 31 01:50:25.714887 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.714770    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52"} err="failed to get container status \"44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52\": rpc error: code = NotFound desc = could not find container \"44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52\": container with ID starting with 44d272101fef3c75b4d5a1d4016557e59a4f2496850b0b80014193aa339a9b52 not found: ID does not exist"
Oct 31 01:50:25.745316 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.745288    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:25.745316 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.745316    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:25.745480 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.745327    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8cln\" (UniqueName: \"kubernetes.io/projected/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-kube-api-access-c8cln\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:25.745480 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.745338    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f90bdfe8-cc91-4f35-973b-ca7f921e2f31-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:25.865243 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.865139    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"]
Oct 31 01:50:25.869321 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.869282    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8a3796-657d78d597-jwpkp"]
Oct 31 01:50:25.969152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:25.969120    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90bdfe8-cc91-4f35-973b-ca7f921e2f31" path="/var/lib/kubelet/pods/f90bdfe8-cc91-4f35-973b-ca7f921e2f31/volumes"
Oct 31 01:50:26.390754 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.390719    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"]
Oct 31 01:50:26.391251 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.391020    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" podUID="bd418b2e-650b-438a-899a-9dbe22e92099" containerName="maas-api" containerID="cri-o://bad1ea84f3149010b5e33aae1fca1d998a4942b572e54112e9f33462aa9f662a" gracePeriod=30
Oct 31 01:50:26.711016 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.710975    5718 generic.go:356] "Generic (PLEG): container finished" podID="bd418b2e-650b-438a-899a-9dbe22e92099" containerID="bad1ea84f3149010b5e33aae1fca1d998a4942b572e54112e9f33462aa9f662a" exitCode=0
Oct 31 01:50:26.711188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.711106    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" event={"ID":"bd418b2e-650b-438a-899a-9dbe22e92099","Type":"ContainerDied","Data":"bad1ea84f3149010b5e33aae1fca1d998a4942b572e54112e9f33462aa9f662a"}
Oct 31 01:50:26.711188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.711160    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm" event={"ID":"bd418b2e-650b-438a-899a-9dbe22e92099","Type":"ContainerDied","Data":"e1cae93c48d4f844608ab4955089dee68d3ee4309998a3e6bb34256c305b2287"}
Oct 31 01:50:26.711188 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.711185    5718 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1cae93c48d4f844608ab4955089dee68d3ee4309998a3e6bb34256c305b2287"
Oct 31 01:50:26.726585 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.726557    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:50:26.858923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.858873    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca\") pod \"bd418b2e-650b-438a-899a-9dbe22e92099\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") "
Oct 31 01:50:26.859090 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.858937    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\") pod \"bd418b2e-650b-438a-899a-9dbe22e92099\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") "
Oct 31 01:50:26.859152 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.859135    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn\" (UniqueName: \"kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn\") pod \"bd418b2e-650b-438a-899a-9dbe22e92099\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") "
Oct 31 01:50:26.859214 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.859198    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs\") pod \"bd418b2e-650b-438a-899a-9dbe22e92099\" (UID: \"bd418b2e-650b-438a-899a-9dbe22e92099\") "
Oct 31 01:50:26.859759 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.859718    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca" pod "bd418b2e-650b-438a-899a-9dbe22e92099" (UID: "bd418b2e-650b-438a-899a-9dbe22e92099"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:26.859935 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.859906    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd418b2e-650b-438a-899a-9dbe22e92099-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:26.861999 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.861973    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs" pod "bd418b2e-650b-438a-899a-9dbe22e92099" (UID: "bd418b2e-650b-438a-899a-9dbe22e92099"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:26.862168 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.862148    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn" pod "bd418b2e-650b-438a-899a-9dbe22e92099" (UID: "bd418b2e-650b-438a-899a-9dbe22e92099"). InnerVolumeSpecName "kube-api-access-29mpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:26.864141 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.864118    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls" pod "bd418b2e-650b-438a-899a-9dbe22e92099" (UID: "bd418b2e-650b-438a-899a-9dbe22e92099"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:26.961519 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.961436    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:26.961699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.961558    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29mpn\" (UniqueName: \"kubernetes.io/projected/bd418b2e-650b-438a-899a-9dbe22e92099-kube-api-access-29mpn\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:26.961699 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:26.961573    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd418b2e-650b-438a-899a-9dbe22e92099-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:27.718141 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:27.718106    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"
Oct 31 01:50:27.888924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:27.888879    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"]
Oct 31 01:50:27.941033 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:27.940998    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c116f3-545b567b59-tm2qm"]
Oct 31 01:50:27.968639 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:27.968544    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd418b2e-650b-438a-899a-9dbe22e92099" path="/var/lib/kubelet/pods/bd418b2e-650b-438a-899a-9dbe22e92099/volumes"
Oct 31 01:50:35.136941 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:35.136904    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"]
Oct 31 01:50:35.137512 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:35.137125    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" containerID="cri-o://749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b" gracePeriod=30
Oct 31 01:50:35.147434 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:35.147406    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"]
Oct 31 01:50:35.147641 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:35.147621    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" containerID="cri-o://a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c" gracePeriod=30
Oct 31 01:50:35.384213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:35.384174    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"]
Oct 31 01:50:37.925570 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:50:37.925524    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:50:39.873416 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:39.873371    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:39.873830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:39.873458    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:40.107442 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.107402    5718 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:40.107633 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.107477    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:40.425282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.425215    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:50:40.449700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.449642    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:50:40.533554 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533522    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.533730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533567    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.533730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533595    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.533730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533627    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.533730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533691    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.533730 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533721    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533758    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533787    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533833    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533871    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533964    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533979    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533996    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.533989    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534011    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534080 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534052    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534164    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534213    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534219    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy\") pod \"0884023c-eaf6-413e-aded-abdfb59c064f\" (UID: \"0884023c-eaf6-413e-aded-abdfb59c064f\") "
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534266    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534343    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534395    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket\") pod \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\" (UID: \"0cdda880-ef9c-4c59-aa6f-52640e0652ed\") "
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534218    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534386    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534453    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:40.534793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534755    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.535311 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.534955    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535472    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535500    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535516    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535531    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535546    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535560    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535575    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0884023c-eaf6-413e-aded-abdfb59c064f-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535589    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535603    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536172 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.535932    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:40.536819 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.536797    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "kube-api-access-d57zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537084 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537064    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:40.537156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537095    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "kube-api-access-7kjh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537219    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537296 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537237    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:40.537520 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537499    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537610 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537547    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy" pod "0884023c-eaf6-413e-aded-abdfb59c064f" (UID: "0884023c-eaf6-413e-aded-abdfb59c064f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.537907 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.537894    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy" pod "0cdda880-ef9c-4c59-aa6f-52640e0652ed" (UID: "0cdda880-ef9c-4c59-aa6f-52640e0652ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.637254 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637220    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637254 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637251    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637254 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637260    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0884023c-eaf6-413e-aded-abdfb59c064f-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637268    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637279    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637290    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57zf\" (UniqueName: \"kubernetes.io/projected/0884023c-eaf6-413e-aded-abdfb59c064f-kube-api-access-d57zf\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637300    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdda880-ef9c-4c59-aa6f-52640e0652ed-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637308    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjh9\" (UniqueName: \"kubernetes.io/projected/0cdda880-ef9c-4c59-aa6f-52640e0652ed-kube-api-access-7kjh9\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637502 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.637317    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0884023c-eaf6-413e-aded-abdfb59c064f-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.805558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.805526    5718 generic.go:356] "Generic (PLEG): container finished" podID="0884023c-eaf6-413e-aded-abdfb59c064f" containerID="a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c" exitCode=0
Oct 31 01:50:40.805804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.805585    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"
Oct 31 01:50:40.805804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.805615    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" event={"ID":"0884023c-eaf6-413e-aded-abdfb59c064f","Type":"ContainerDied","Data":"a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c"}
Oct 31 01:50:40.805804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.805718    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt" event={"ID":"0884023c-eaf6-413e-aded-abdfb59c064f","Type":"ContainerDied","Data":"5b3887549c0ee93551b8038cf6b31b7f5ff6a8978bb2e212c21c2389d37f6e2b"}
Oct 31 01:50:40.805804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.805738    5718 scope.go:122] "RemoveContainer" containerID="a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c"
Oct 31 01:50:40.807222 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.807200    5718 generic.go:356] "Generic (PLEG): container finished" podID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerID="749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b" exitCode=0
Oct 31 01:50:40.807340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.807238    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" event={"ID":"0cdda880-ef9c-4c59-aa6f-52640e0652ed","Type":"ContainerDied","Data":"749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b"}
Oct 31 01:50:40.807340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.807279    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx" event={"ID":"0cdda880-ef9c-4c59-aa6f-52640e0652ed","Type":"ContainerDied","Data":"73260b2f7864969ddf2537801ec259e5c4576f65c0dcd729e9295afb3137a9cd"}
Oct 31 01:50:40.807340 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.807244    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"
Oct 31 01:50:40.817312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.817292    5718 scope.go:122] "RemoveContainer" containerID="a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c"
Oct 31 01:50:40.817608 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:50:40.817588    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c\": container with ID starting with a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c not found: ID does not exist" containerID="a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c"
Oct 31 01:50:40.817701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.817617    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c"} err="failed to get container status \"a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c\": rpc error: code = NotFound desc = could not find container \"a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c\": container with ID starting with a4c6b7b321186290af95b07fe9b0e112d20b41e1e67fbe8a33f89cb08f5cf59c not found: ID does not exist"
Oct 31 01:50:40.817701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.817637    5718 scope.go:122] "RemoveContainer" containerID="749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b"
Oct 31 01:50:40.826882 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.826868    5718 scope.go:122] "RemoveContainer" containerID="749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b"
Oct 31 01:50:40.827153 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:50:40.827136    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b\": container with ID starting with 749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b not found: ID does not exist" containerID="749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b"
Oct 31 01:50:40.827192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.827161    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b"} err="failed to get container status \"749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b\": rpc error: code = NotFound desc = could not find container \"749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b\": container with ID starting with 749fc7b37c49f3d1fed845a51e27d34e0a1dc25381db93d6b5ee42fdbb5aa17b not found: ID does not exist"
Oct 31 01:50:40.975579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.975545    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"]
Oct 31 01:50:40.978910 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:40.978882    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2m7dx"]
Oct 31 01:50:41.091901 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:41.091812    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"]
Oct 31 01:50:41.095298 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:41.095270    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-b8lrt"]
Oct 31 01:50:41.967418 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:41.967387    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" path="/var/lib/kubelet/pods/0884023c-eaf6-413e-aded-abdfb59c064f/volumes"
Oct 31 01:50:41.967897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:41.967882    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" path="/var/lib/kubelet/pods/0cdda880-ef9c-4c59-aa6f-52640e0652ed/volumes"
Oct 31 01:50:50.407551 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.407501    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj" podUID="99412668-a12c-4264-b97c-38166feb93ab" containerName="main" containerID="cri-o://0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c" gracePeriod=45
Oct 31 01:50:50.669885 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.669821    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:50:50.868804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.868771    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir\") pod \"99412668-a12c-4264-b97c-38166feb93ab\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") "
Oct 31 01:50:50.868804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.868810    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home\") pod \"99412668-a12c-4264-b97c-38166feb93ab\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") "
Oct 31 01:50:50.869070 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.868858    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache\") pod \"99412668-a12c-4264-b97c-38166feb93ab\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") "
Oct 31 01:50:50.869070 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.868891    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs\") pod \"99412668-a12c-4264-b97c-38166feb93ab\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") "
Oct 31 01:50:50.869070 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.868923    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm\" (UniqueName: \"kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm\") pod \"99412668-a12c-4264-b97c-38166feb93ab\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") "
Oct 31 01:50:50.869070 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.868957    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm\") pod \"99412668-a12c-4264-b97c-38166feb93ab\" (UID: \"99412668-a12c-4264-b97c-38166feb93ab\") "
Oct 31 01:50:50.869286 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.869088    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home" pod "99412668-a12c-4264-b97c-38166feb93ab" (UID: "99412668-a12c-4264-b97c-38166feb93ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:50.869286 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.869095    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir" pod "99412668-a12c-4264-b97c-38166feb93ab" (UID: "99412668-a12c-4264-b97c-38166feb93ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:50.869398 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.869305    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache" pod "99412668-a12c-4264-b97c-38166feb93ab" (UID: "99412668-a12c-4264-b97c-38166feb93ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:50.869503 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.869480    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.869580 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.869507    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.869580 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.869521    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.870435 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.870401    5718 generic.go:356] "Generic (PLEG): container finished" podID="99412668-a12c-4264-b97c-38166feb93ab" containerID="0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c" exitCode=0
Oct 31 01:50:50.870637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.870455    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"
Oct 31 01:50:50.870637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.870486    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj" event={"ID":"99412668-a12c-4264-b97c-38166feb93ab","Type":"ContainerDied","Data":"0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c"}
Oct 31 01:50:50.870637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.870540    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj" event={"ID":"99412668-a12c-4264-b97c-38166feb93ab","Type":"ContainerDied","Data":"95c4f9ab84d20c6fe2a21dba4f031be356205a4f406175173db46507456c36fc"}
Oct 31 01:50:50.870637 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.870564    5718 scope.go:122] "RemoveContainer" containerID="0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c"
Oct 31 01:50:50.871815 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.871794    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm" pod "99412668-a12c-4264-b97c-38166feb93ab" (UID: "99412668-a12c-4264-b97c-38166feb93ab"). InnerVolumeSpecName "kube-api-access-rh8sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:50.871909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.871891    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm" pod "99412668-a12c-4264-b97c-38166feb93ab" (UID: "99412668-a12c-4264-b97c-38166feb93ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:50.871973 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.871959    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs" pod "99412668-a12c-4264-b97c-38166feb93ab" (UID: "99412668-a12c-4264-b97c-38166feb93ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:50.898474 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.896413    5718 scope.go:122] "RemoveContainer" containerID="0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c"
Oct 31 01:50:50.898474 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:50:50.896815    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c\": container with ID starting with 0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c not found: ID does not exist" containerID="0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c"
Oct 31 01:50:50.898474 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.896846    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c"} err="failed to get container status \"0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c\": rpc error: code = NotFound desc = could not find container \"0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c\": container with ID starting with 0928fe01775230c985183667cee31be9781a38a1ffa7df90090a0376bb932c0c not found: ID does not exist"
Oct 31 01:50:50.971115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.971039    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99412668-a12c-4264-b97c-38166feb93ab-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.971115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.971064    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8sm\" (UniqueName: \"kubernetes.io/projected/99412668-a12c-4264-b97c-38166feb93ab-kube-api-access-rh8sm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.971115 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:50.971073    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99412668-a12c-4264-b97c-38166feb93ab-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:51.293324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:51.293290    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"]
Oct 31 01:50:51.343312 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:51.343283    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-53af08/ipp-route-53af08-kserve-cd76b4477-h64wj"]
Oct 31 01:50:51.967511 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:50:51.967467    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99412668-a12c-4264-b97c-38166feb93ab" path="/var/lib/kubelet/pods/99412668-a12c-4264-b97c-38166feb93ab/volumes"
Oct 31 01:51:07.924441 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:51:07.924401    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:51:24.912969 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:24.912875    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"]
Oct 31 01:51:24.913529 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:24.913092    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" podUID="cf7fa906-e80d-49f4-972f-e635136cccb3" containerName="maas-api" containerID="cri-o://d7e4cd50d6240d0453d31bf0586bdf784fe2142082650130260e5c5e570c2cf0" gracePeriod=30
Oct 31 01:51:25.073901 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.073866    5718 generic.go:356] "Generic (PLEG): container finished" podID="cf7fa906-e80d-49f4-972f-e635136cccb3" containerID="d7e4cd50d6240d0453d31bf0586bdf784fe2142082650130260e5c5e570c2cf0" exitCode=0
Oct 31 01:51:25.074077 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.073944    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" event={"ID":"cf7fa906-e80d-49f4-972f-e635136cccb3","Type":"ContainerDied","Data":"d7e4cd50d6240d0453d31bf0586bdf784fe2142082650130260e5c5e570c2cf0"}
Oct 31 01:51:25.218297 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.218268    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:51:25.275545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.275505    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca\") pod \"cf7fa906-e80d-49f4-972f-e635136cccb3\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") "
Oct 31 01:51:25.275777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.275557    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\") pod \"cf7fa906-e80d-49f4-972f-e635136cccb3\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") "
Oct 31 01:51:25.275777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.275625    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs\") pod \"cf7fa906-e80d-49f4-972f-e635136cccb3\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") "
Oct 31 01:51:25.275777 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.275740    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7\" (UniqueName: \"kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7\") pod \"cf7fa906-e80d-49f4-972f-e635136cccb3\" (UID: \"cf7fa906-e80d-49f4-972f-e635136cccb3\") "
Oct 31 01:51:25.275979 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.275942    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca" pod "cf7fa906-e80d-49f4-972f-e635136cccb3" (UID: "cf7fa906-e80d-49f4-972f-e635136cccb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:25.276302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.276262    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7fa906-e80d-49f4-972f-e635136cccb3-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:25.278058 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.278024    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7" pod "cf7fa906-e80d-49f4-972f-e635136cccb3" (UID: "cf7fa906-e80d-49f4-972f-e635136cccb3"). InnerVolumeSpecName "kube-api-access-x7ng7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:25.278192 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.278080    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs" pod "cf7fa906-e80d-49f4-972f-e635136cccb3" (UID: "cf7fa906-e80d-49f4-972f-e635136cccb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:25.278281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.278257    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls" pod "cf7fa906-e80d-49f4-972f-e635136cccb3" (UID: "cf7fa906-e80d-49f4-972f-e635136cccb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:25.377742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.377701    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:25.377742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.377738    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7fa906-e80d-49f4-972f-e635136cccb3-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:25.377742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:25.377749    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ng7\" (UniqueName: \"kubernetes.io/projected/cf7fa906-e80d-49f4-972f-e635136cccb3-kube-api-access-x7ng7\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:26.080558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:26.080526    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"
Oct 31 01:51:26.081034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:26.080551    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw" event={"ID":"cf7fa906-e80d-49f4-972f-e635136cccb3","Type":"ContainerDied","Data":"5fcf7b0df24aa661f412974b3dceff30864205d1c2e2e02283e2de543407487f"}
Oct 31 01:51:26.081034 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:26.080627    5718 scope.go:122] "RemoveContainer" containerID="d7e4cd50d6240d0453d31bf0586bdf784fe2142082650130260e5c5e570c2cf0"
Oct 31 01:51:26.288827 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:26.288754    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"]
Oct 31 01:51:26.338683 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:26.338635    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9d7b2c-656586984-5nwvw"]
Oct 31 01:51:27.967902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:27.967868    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7fa906-e80d-49f4-972f-e635136cccb3" path="/var/lib/kubelet/pods/cf7fa906-e80d-49f4-972f-e635136cccb3/volumes"
Oct 31 01:51:37.924720 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:51:37.924643    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:51:41.182829 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.182794    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"]
Oct 31 01:51:41.183282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.183009    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" podUID="6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" containerName="maas-api" containerID="cri-o://bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5" gracePeriod=30
Oct 31 01:51:41.508286 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.508248    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:51:41.661227 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.661191    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph\" (UniqueName: \"kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph\") pod \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") "
Oct 31 01:51:41.661423 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.661312    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca\") pod \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") "
Oct 31 01:51:41.661504 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.661428    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs\") pod \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") "
Oct 31 01:51:41.661560 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.661502    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\") pod \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\" (UID: \"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541\") "
Oct 31 01:51:41.661731 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.661703    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca" pod "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" (UID: "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:41.662127 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.662033    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:41.663679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.663641    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs" pod "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" (UID: "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:41.664178 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.664150    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls" pod "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" (UID: "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:41.664178 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.664167    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph" pod "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" (UID: "6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541"). InnerVolumeSpecName "kube-api-access-stnph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:41.763136 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.763098    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:41.763136 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.763133    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:41.763354 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:41.763146    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnph\" (UniqueName: \"kubernetes.io/projected/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541-kube-api-access-stnph\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:42.190460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.190360    5718 generic.go:356] "Generic (PLEG): container finished" podID="6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" containerID="bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5" exitCode=0
Oct 31 01:51:42.190460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.190413    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"
Oct 31 01:51:42.190460 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.190443    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" event={"ID":"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541","Type":"ContainerDied","Data":"bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5"}
Oct 31 01:51:42.190990 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.190506    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr" event={"ID":"6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541","Type":"ContainerDied","Data":"3ab5e3db7cee27dd452d00b89336690434518c6250a9d8e00e5d32cc55b1732d"}
Oct 31 01:51:42.190990 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.190531    5718 scope.go:122] "RemoveContainer" containerID="bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5"
Oct 31 01:51:42.200767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.200747    5718 scope.go:122] "RemoveContainer" containerID="bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5"
Oct 31 01:51:42.201052 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:51:42.201034    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5\": container with ID starting with bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5 not found: ID does not exist" containerID="bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5"
Oct 31 01:51:42.201095 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.201062    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5"} err="failed to get container status \"bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5\": rpc error: code = NotFound desc = could not find container \"bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5\": container with ID starting with bf293d11c962a5c37dd9880feed36368e2a494a19b34200120bace7052189af5 not found: ID does not exist"
Oct 31 01:51:42.355226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.355188    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"]
Oct 31 01:51:42.405620 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:42.405588    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e59f12-747cf6c46b-fz2cr"]
Oct 31 01:51:43.967562 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:43.967532    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" path="/var/lib/kubelet/pods/6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541/volumes"
Oct 31 01:51:44.594736 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:44.594703    5718 scope.go:122] "RemoveContainer" containerID="bad1ea84f3149010b5e33aae1fca1d998a4942b572e54112e9f33462aa9f662a"
Oct 31 01:51:57.780836 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:57.780802    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"]
Oct 31 01:51:57.781316 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:57.781075    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" podUID="3b40ca9c-6716-4f94-9719-5313ef504b5a" containerName="maas-api" containerID="cri-o://d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378" gracePeriod=30
Oct 31 01:51:58.074284 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.074250    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:51:58.152669 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.152607    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p\" (UniqueName: \"kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p\") pod \"3b40ca9c-6716-4f94-9719-5313ef504b5a\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") "
Oct 31 01:51:58.152900 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.152776    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs\") pod \"3b40ca9c-6716-4f94-9719-5313ef504b5a\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") "
Oct 31 01:51:58.152984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.152924    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca\") pod \"3b40ca9c-6716-4f94-9719-5313ef504b5a\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") "
Oct 31 01:51:58.152984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.152964    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\") pod \"3b40ca9c-6716-4f94-9719-5313ef504b5a\" (UID: \"3b40ca9c-6716-4f94-9719-5313ef504b5a\") "
Oct 31 01:51:58.153333 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.153296    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca" pod "3b40ca9c-6716-4f94-9719-5313ef504b5a" (UID: "3b40ca9c-6716-4f94-9719-5313ef504b5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:58.153490 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.153469    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b40ca9c-6716-4f94-9719-5313ef504b5a-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.155212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.155190    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs" pod "3b40ca9c-6716-4f94-9719-5313ef504b5a" (UID: "3b40ca9c-6716-4f94-9719-5313ef504b5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:58.155448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.155422    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p" pod "3b40ca9c-6716-4f94-9719-5313ef504b5a" (UID: "3b40ca9c-6716-4f94-9719-5313ef504b5a"). InnerVolumeSpecName "kube-api-access-vkv4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:58.155579 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.155559    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls" pod "3b40ca9c-6716-4f94-9719-5313ef504b5a" (UID: "3b40ca9c-6716-4f94-9719-5313ef504b5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:58.254573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.254539    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.254573 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.254572    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkv4p\" (UniqueName: \"kubernetes.io/projected/3b40ca9c-6716-4f94-9719-5313ef504b5a-kube-api-access-vkv4p\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.254857 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.254585    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b40ca9c-6716-4f94-9719-5313ef504b5a-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:58.286538 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.286498    5718 generic.go:356] "Generic (PLEG): container finished" podID="3b40ca9c-6716-4f94-9719-5313ef504b5a" containerID="d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378" exitCode=0
Oct 31 01:51:58.286758 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.286550    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"
Oct 31 01:51:58.286758 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.286554    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" event={"ID":"3b40ca9c-6716-4f94-9719-5313ef504b5a","Type":"ContainerDied","Data":"d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378"}
Oct 31 01:51:58.286758 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.286626    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z" event={"ID":"3b40ca9c-6716-4f94-9719-5313ef504b5a","Type":"ContainerDied","Data":"d95e43219730af64fd438f1ebe82a25744e5e491cd84607aad47bdf9db340df0"}
Oct 31 01:51:58.286758 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.286678    5718 scope.go:122] "RemoveContainer" containerID="d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378"
Oct 31 01:51:58.297609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.297591    5718 scope.go:122] "RemoveContainer" containerID="d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378"
Oct 31 01:51:58.297889 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:51:58.297864    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378\": container with ID starting with d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378 not found: ID does not exist" containerID="d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378"
Oct 31 01:51:58.297957 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.297892    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378"} err="failed to get container status \"d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378\": rpc error: code = NotFound desc = could not find container \"d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378\": container with ID starting with d9be13408135f03580e934b89a80316fed9a2e59ce99fec506e7effba47ea378 not found: ID does not exist"
Oct 31 01:51:58.455213 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.455176    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"]
Oct 31 01:51:58.461369 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:58.461337    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99145d-75c8f559fd-srw7z"]
Oct 31 01:51:59.967903 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:51:59.967865    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b40ca9c-6716-4f94-9719-5313ef504b5a" path="/var/lib/kubelet/pods/3b40ca9c-6716-4f94-9719-5313ef504b5a/volumes"
Oct 31 01:52:07.924744 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:52:07.924704    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:52:13.803905 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:13.803867    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"]
Oct 31 01:52:13.804437 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:13.804136    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" podUID="cc40539a-0207-4302-83d6-e958d63a76a4" containerName="maas-api" containerID="cri-o://e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5" gracePeriod=30
Oct 31 01:52:14.107421 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.107387    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:52:14.258195 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.258162    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs\") pod \"cc40539a-0207-4302-83d6-e958d63a76a4\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") "
Oct 31 01:52:14.258412 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.258230    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\") pod \"cc40539a-0207-4302-83d6-e958d63a76a4\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") "
Oct 31 01:52:14.258412 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.258340    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca\") pod \"cc40539a-0207-4302-83d6-e958d63a76a4\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") "
Oct 31 01:52:14.258547 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.258441    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj\") pod \"cc40539a-0207-4302-83d6-e958d63a76a4\" (UID: \"cc40539a-0207-4302-83d6-e958d63a76a4\") "
Oct 31 01:52:14.258983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.258915    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca" pod "cc40539a-0207-4302-83d6-e958d63a76a4" (UID: "cc40539a-0207-4302-83d6-e958d63a76a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:14.260915 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.260892    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs" pod "cc40539a-0207-4302-83d6-e958d63a76a4" (UID: "cc40539a-0207-4302-83d6-e958d63a76a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:14.261767 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.261736    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls" pod "cc40539a-0207-4302-83d6-e958d63a76a4" (UID: "cc40539a-0207-4302-83d6-e958d63a76a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:14.261891 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.261870    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj" pod "cc40539a-0207-4302-83d6-e958d63a76a4" (UID: "cc40539a-0207-4302-83d6-e958d63a76a4"). InnerVolumeSpecName "kube-api-access-bbktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:14.313852 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.313812    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"]
Oct 31 01:52:14.314101 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.314065    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw" podUID="0ce34f90-44db-410b-907c-51433f1b5a49" containerName="payload-pre-processing" containerID="cri-o://a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f" gracePeriod=30
Oct 31 01:52:14.359806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.359712    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbktj\" (UniqueName: \"kubernetes.io/projected/cc40539a-0207-4302-83d6-e958d63a76a4-kube-api-access-bbktj\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:14.359806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.359748    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:14.359806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.359762    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40539a-0207-4302-83d6-e958d63a76a4-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:14.359806 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.359779    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40539a-0207-4302-83d6-e958d63a76a4-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:14.380221 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.380187    5718 generic.go:356] "Generic (PLEG): container finished" podID="cc40539a-0207-4302-83d6-e958d63a76a4" containerID="e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5" exitCode=0
Oct 31 01:52:14.380407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.380239    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"
Oct 31 01:52:14.380407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.380262    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" event={"ID":"cc40539a-0207-4302-83d6-e958d63a76a4","Type":"ContainerDied","Data":"e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5"}
Oct 31 01:52:14.380407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.380324    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj" event={"ID":"cc40539a-0207-4302-83d6-e958d63a76a4","Type":"ContainerDied","Data":"004cf2f172d920b2e80ae6ebf7d5ab6042eaa16bde805df95f66d691925d15e1"}
Oct 31 01:52:14.380407 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.380359    5718 scope.go:122] "RemoveContainer" containerID="e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5"
Oct 31 01:52:14.433294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.432839    5718 scope.go:122] "RemoveContainer" containerID="e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5"
Oct 31 01:52:14.433294 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:52:14.433194    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5\": container with ID starting with e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5 not found: ID does not exist" containerID="e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5"
Oct 31 01:52:14.433294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.433236    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5"} err="failed to get container status \"e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5\": rpc error: code = NotFound desc = could not find container \"e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5\": container with ID starting with e8078e21be69d4e785f34b8c558840aeebf2c071f38e1ea3271c9743d67f16d5 not found: ID does not exist"
Oct 31 01:52:14.545415 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.545383    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"]
Oct 31 01:52:14.547360 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.547334    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b2d49b-576b5bcb7f-9kxhj"]
Oct 31 01:52:14.573262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.573229    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:52:14.662802 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.662716    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh\" (UniqueName: \"kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh\") pod \"0ce34f90-44db-410b-907c-51433f1b5a49\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") "
Oct 31 01:52:14.665060 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.665034    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh" pod "0ce34f90-44db-410b-907c-51433f1b5a49" (UID: "0ce34f90-44db-410b-907c-51433f1b5a49"). InnerVolumeSpecName "kube-api-access-msbkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:14.763716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.763677    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume\") pod \"0ce34f90-44db-410b-907c-51433f1b5a49\" (UID: \"0ce34f90-44db-410b-907c-51433f1b5a49\") "
Oct 31 01:52:14.764118 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.764083    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbkh\" (UniqueName: \"kubernetes.io/projected/0ce34f90-44db-410b-907c-51433f1b5a49-kube-api-access-msbkh\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:14.769764 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.765052    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume" pod "0ce34f90-44db-410b-907c-51433f1b5a49" (UID: "0ce34f90-44db-410b-907c-51433f1b5a49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:14.864968 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:14.864933    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce34f90-44db-410b-907c-51433f1b5a49-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:15.388801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.388758    5718 generic.go:356] "Generic (PLEG): container finished" podID="0ce34f90-44db-410b-907c-51433f1b5a49" containerID="a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f" exitCode=0
Oct 31 01:52:15.389123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.388808    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"
Oct 31 01:52:15.389123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.388841    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw" event={"ID":"0ce34f90-44db-410b-907c-51433f1b5a49","Type":"ContainerDied","Data":"a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f"}
Oct 31 01:52:15.389123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.388896    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw" event={"ID":"0ce34f90-44db-410b-907c-51433f1b5a49","Type":"ContainerDied","Data":"203793cfedb8b7618ec00b9f23a4ef6f4a4a5e982243661760d03c6dc41d1336"}
Oct 31 01:52:15.389123 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.388919    5718 scope.go:122] "RemoveContainer" containerID="a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f"
Oct 31 01:52:15.398902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.398883    5718 scope.go:122] "RemoveContainer" containerID="a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f"
Oct 31 01:52:15.399189 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:52:15.399169    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f\": container with ID starting with a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f not found: ID does not exist" containerID="a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f"
Oct 31 01:52:15.399237 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.399199    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f"} err="failed to get container status \"a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f\": rpc error: code = NotFound desc = could not find container \"a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f\": container with ID starting with a7def0e70f8bc6d1f842e7800f8ff0eff483be72f7e72b24c1aafe3b6ef7dc5f not found: ID does not exist"
Oct 31 01:52:15.553046 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.553012    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"]
Oct 31 01:52:15.557160 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.557123    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-79df5496bc-698bw"]
Oct 31 01:52:15.967165 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.967125    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce34f90-44db-410b-907c-51433f1b5a49" path="/var/lib/kubelet/pods/0ce34f90-44db-410b-907c-51433f1b5a49/volumes"
Oct 31 01:52:15.967800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:52:15.967780    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc40539a-0207-4302-83d6-e958d63a76a4" path="/var/lib/kubelet/pods/cc40539a-0207-4302-83d6-e958d63a76a4/volumes"
Oct 31 01:52:37.924441 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:52:37.924399    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:53:07.923404 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:53:07.923316    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:53:08.023733 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.023698    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"]
Oct 31 01:53:08.024379 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024348    5718 cpu_manager.go:414] "containerMap: removing container" podUID="0ce34f90-44db-410b-907c-51433f1b5a49" containerName="payload-pre-processing"
Oct 31 01:53:08.024379 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024378    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce34f90-44db-410b-907c-51433f1b5a49" containerName="payload-pre-processing"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024397    5718 cpu_manager.go:414] "containerMap: removing container" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024404    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024417    5718 cpu_manager.go:414] "containerMap: removing container" podUID="f90bdfe8-cc91-4f35-973b-ca7f921e2f31" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024422    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90bdfe8-cc91-4f35-973b-ca7f921e2f31" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024436    5718 cpu_manager.go:414] "containerMap: removing container" podUID="cc40539a-0207-4302-83d6-e958d63a76a4" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024441    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc40539a-0207-4302-83d6-e958d63a76a4" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024453    5718 cpu_manager.go:414] "containerMap: removing container" podUID="cf7fa906-e80d-49f4-972f-e635136cccb3" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024457    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7fa906-e80d-49f4-972f-e635136cccb3" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024473    5718 cpu_manager.go:414] "containerMap: removing container" podUID="bd418b2e-650b-438a-899a-9dbe22e92099" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024478    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd418b2e-650b-438a-899a-9dbe22e92099" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024489    5718 cpu_manager.go:414] "containerMap: removing container" podUID="99412668-a12c-4264-b97c-38166feb93ab" containerName="main"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024493    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99412668-a12c-4264-b97c-38166feb93ab" containerName="main"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024498    5718 cpu_manager.go:414] "containerMap: removing container" podUID="3b40ca9c-6716-4f94-9719-5313ef504b5a" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024503    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b40ca9c-6716-4f94-9719-5313ef504b5a" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024512    5718 cpu_manager.go:414] "containerMap: removing container" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024516    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024525    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" containerName="maas-api"
Oct 31 01:53:08.024555 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024530    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" containerName="maas-api"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024619    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce34f90-44db-410b-907c-51433f1b5a49" containerName="payload-pre-processing"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024629    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd418b2e-650b-438a-899a-9dbe22e92099" containerName="maas-api"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024634    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="0884023c-eaf6-413e-aded-abdfb59c064f" containerName="istio-proxy"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024642    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90bdfe8-cc91-4f35-973b-ca7f921e2f31" containerName="maas-api"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024668    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="99412668-a12c-4264-b97c-38166feb93ab" containerName="main"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024674    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b40ca9c-6716-4f94-9719-5313ef504b5a" containerName="maas-api"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024680    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc40539a-0207-4302-83d6-e958d63a76a4" containerName="maas-api"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024686    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdda880-ef9c-4c59-aa6f-52640e0652ed" containerName="istio-proxy"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024693    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfcaa6f-3cc3-4b0f-b3fe-a47ba0eeb541" containerName="maas-api"
Oct 31 01:53:08.025187 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.024698    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7fa906-e80d-49f4-972f-e635136cccb3" containerName="maas-api"
Oct 31 01:53:08.029581 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.029559    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.097530 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097500    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.097720 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097539    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.097720 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097566    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.097720 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097689    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.097898 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097840    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4t5\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.097959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097930    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.098007 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.097969    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.098048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.098022    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.098099 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.098046    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.135487 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.135440    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"]
Oct 31 01:53:08.168444 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.168415    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a8e157-openshift-default-dockercfg-f24hn\""
Oct 31 01:53:08.199048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.198959    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199013    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199048 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199038    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199056    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199175    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199240    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199446    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199569 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199550    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.199664 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199636    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.200082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199868    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.200082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199879    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4t5\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.200082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199633    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.200082 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.199699    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.200279 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.200082    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.201955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.201932    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.202027 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.202002    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.257619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.257574    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.257804 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.257675    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4t5\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.343165 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.343129    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:08.768478 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.768429    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"]
Oct 31 01:53:08.769566 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:53:08.769534    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0a5d9c_4c28_4951_969f_1c6b7cf99b7d.slice/crio-6fffe3de4365db76161c7d976f32406912513a78cc532852a37220ec6a0a40c2 WatchSource:0}: Error finding container 6fffe3de4365db76161c7d976f32406912513a78cc532852a37220ec6a0a40c2: Status 404 returned error can't find the container with id 6fffe3de4365db76161c7d976f32406912513a78cc532852a37220ec6a0a40c2
Oct 31 01:53:08.772102 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.772070    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:08.772157 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.772137    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:08.772193 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:08.772164    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:09.704694 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:09.704628    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" event={"ID":"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d","Type":"ContainerStarted","Data":"5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e"}
Oct 31 01:53:09.705156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:09.704716    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" event={"ID":"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d","Type":"ContainerStarted","Data":"6fffe3de4365db76161c7d976f32406912513a78cc532852a37220ec6a0a40c2"}
Oct 31 01:53:09.877924 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:09.877869    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podStartSLOduration=2.877851587 podStartE2EDuration="2.877851587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:09.875345336 +0000 UTC m=+2906.038488751" watchObservedRunningTime="2026-10-31 01:53:09.877851587 +0000 UTC m=+2906.040995002"
Oct 31 01:53:10.352684 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:10.348057    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:10.363693 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:10.353739    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:53:10.363693 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:10.353886    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:53:11.360113 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.360072    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:53:11.361764 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.361723    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:53:11.471937 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.471902    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"]
Oct 31 01:53:11.476825 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.476801    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.530702 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.530644    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-00e840-openshift-default-dockercfg-2nfqp\""
Oct 31 01:53:11.566478 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566416    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566497    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566563    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566587    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vb4\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566618    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566716 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566674    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566765    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566803    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.566962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.566833    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.585020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.584981    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"]
Oct 31 01:53:11.668741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668366    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.668741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668474    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.668741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668515    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.668741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668539    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.668741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668622    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.668741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668726    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.669184 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668788    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.669184 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668828    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vb4\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.669184 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.668865    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.669619 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.669594    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.670542 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.670503    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.670776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.670754    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.671223 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.671198    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.671557 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.671524    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.672735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.672710    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.673212 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.673188    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.733913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.726932    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.733913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.728043    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vb4\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:11.791221 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:11.791183    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:12.265915 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.265878    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"]
Oct 31 01:53:12.282717 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:53:12.282684    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43fa747_14b6_4aa0_8684_3cfd3f6fc1c8.slice/crio-275399dab0f4b51383e8e7bd87a9d260278982bccf70631203a1f41bcd4f1371 WatchSource:0}: Error finding container 275399dab0f4b51383e8e7bd87a9d260278982bccf70631203a1f41bcd4f1371: Status 404 returned error can't find the container with id 275399dab0f4b51383e8e7bd87a9d260278982bccf70631203a1f41bcd4f1371
Oct 31 01:53:12.285219 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.285184    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:12.285300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.285258    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:12.285300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.285291    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:12.365839 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.365802    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:53:12.366258 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.365858    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:53:12.730091 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.729995    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" event={"ID":"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8","Type":"ContainerStarted","Data":"295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da"}
Oct 31 01:53:12.730091 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.730063    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" event={"ID":"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8","Type":"ContainerStarted","Data":"275399dab0f4b51383e8e7bd87a9d260278982bccf70631203a1f41bcd4f1371"}
Oct 31 01:53:12.791417 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:12.791389    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:13.344679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:13.344067    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:53:13.344679 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:13.344134    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:53:13.796304 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:13.796263    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:53:13.796742 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:13.796336    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 01:53:14.348682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:14.345122    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:53:14.348682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:14.345193    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:53:14.793137 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:14.793098    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:53:14.793488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:14.793453    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 01:53:15.345678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:15.345122    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:53:15.345678 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:15.345188    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 01:53:15.792142 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:15.792110    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:53:15.792338 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:15.792165    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 01:53:16.792469 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:16.792425    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:53:16.792911 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:16.792496    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 01:53:17.389853 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:17.389798    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:53:17.390079 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:17.389881    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:17.792868 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:17.792829    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:53:17.797956 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:17.797915    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 01:53:18.131799 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:18.131706    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:18.132443 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:18.132417    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:18.133728 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:18.133704    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:53:18.302244 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:18.302194    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podStartSLOduration=7.302180139 podStartE2EDuration="7.302180139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:12.857026113 +0000 UTC m=+2909.020169529" watchObservedRunningTime="2026-10-31 01:53:18.302180139 +0000 UTC m=+2914.465323554"
Oct 31 01:53:18.791589 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:18.791549    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:53:18.791830 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:18.791632    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 01:53:20.833810 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:20.833771    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:53:20.834272 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:20.833843    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:21.289622 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:21.289553    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:21.290495 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:21.290465    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:21.291203 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:21.291180    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 01:53:37.926131 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:53:37.926093    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:53:56.476632 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.476594    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"]
Oct 31 01:53:56.481158 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.481137    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535396 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535356    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535415    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535484    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535567 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535547    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535710 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535612    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535747 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535714    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535794 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535782    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535801    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.535911 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.535898    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5cs\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.587448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.587414    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"]
Oct 31 01:53:56.637324 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637291    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637366    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637384    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637418    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5cs\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637448    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637489    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637526 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637508    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637583    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637634    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.637989 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637939    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.638038 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.637998    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.638038 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.638014    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.638150 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.638078    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.638372 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.638245    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.639984 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.639962    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.640289 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.640266    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.710566 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.710531    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.712599 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.712572    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5cs\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:56.796220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:56.796182    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:57.223535 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:57.223508    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"]
Oct 31 01:53:57.224983 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:53:57.224956    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5a7932_7467_41e5_86df_e2ec9ecd4610.slice/crio-e3744d10ff78127d21f8f3edebd8fab87900b4241019c620fb7d1461c5731820 WatchSource:0}: Error finding container e3744d10ff78127d21f8f3edebd8fab87900b4241019c620fb7d1461c5731820: Status 404 returned error can't find the container with id e3744d10ff78127d21f8f3edebd8fab87900b4241019c620fb7d1461c5731820
Oct 31 01:53:57.227597 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:57.227566    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:57.227722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:57.227631    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:57.227722 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:57.227687    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:53:58.070741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:58.070702    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" event={"ID":"3e5a7932-7467-41e5-86df-e2ec9ecd4610","Type":"ContainerStarted","Data":"26a25672bb38d68636abd6f7ec36823779b04b09fdfdbb9bcbc72229d607d51c"}
Oct 31 01:53:58.071220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:58.070758    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" event={"ID":"3e5a7932-7467-41e5-86df-e2ec9ecd4610","Type":"ContainerStarted","Data":"e3744d10ff78127d21f8f3edebd8fab87900b4241019c620fb7d1461c5731820"}
Oct 31 01:53:58.291741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:58.291679    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podStartSLOduration=2.291646355 podStartE2EDuration="2.291646355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:58.239733714 +0000 UTC m=+2954.402877129" watchObservedRunningTime="2026-10-31 01:53:58.291646355 +0000 UTC m=+2954.454789773"
Oct 31 01:53:58.796603 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:58.796572    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:53:58.798553 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:58.798529    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:53:58.798630 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:58.798584    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 01:53:59.796809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:59.796773    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:53:59.797217 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:53:59.796829    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 01:54:00.796666 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:00.796613    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:54:00.796859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:00.796728    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 01:54:01.797262 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:01.797223    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:54:01.797776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:01.797293    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 01:54:02.796594 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:02.796565    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:54:02.796872 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:02.796618    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 01:54:03.797377 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:03.797342    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 01:54:03.797798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:03.797400    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 01:54:04.801739 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:04.801711    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:54:04.802129 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:04.802087    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:54:04.802793 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:04.802768    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 01:54:07.926116 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:54:07.926071    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:54:08.487954 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.487918    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"]
Oct 31 01:54:08.492404 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.492380    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.598809 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.598777    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.598829    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599017 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.598955    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599151 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.599024    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599151 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.599059    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599151 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.599141    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjgs\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.599184    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.599212    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.599275 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.599230    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.612317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.612288    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"]
Oct 31 01:54:08.700232 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700196    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700422 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700255    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700422 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700288    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700509 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700450    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjgs\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700544 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700529    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700578 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700568    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700635 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700601    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700753 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700714    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700753 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700732    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700761    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.700859 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.700825    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.701054 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.701024    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.701202 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.701150    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.701348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.701322    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.703281 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.703260    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.703598 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.703575    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.759022 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.758988    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.759241 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.759215    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjgs\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:08.811761 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:08.811724    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:09.242300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:09.242266    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"]
Oct 31 01:54:09.242850 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:54:09.242815    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a56e246_18a3_4aae_a647_0329714ec549.slice/crio-3d8e23a8cced7d88a3a8be04bfd109ec8e6cab4da654c03814858c11030fa350 WatchSource:0}: Error finding container 3d8e23a8cced7d88a3a8be04bfd109ec8e6cab4da654c03814858c11030fa350: Status 404 returned error can't find the container with id 3d8e23a8cced7d88a3a8be04bfd109ec8e6cab4da654c03814858c11030fa350
Oct 31 01:54:09.245524 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:09.245492    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:54:09.245613 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:09.245596    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:54:09.245676 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:09.245640    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:54:10.166572 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:10.166522    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" event={"ID":"5a56e246-18a3-4aae-a647-0329714ec549","Type":"ContainerStarted","Data":"55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d"}
Oct 31 01:54:10.166818 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:10.166602    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" event={"ID":"5a56e246-18a3-4aae-a647-0329714ec549","Type":"ContainerStarted","Data":"3d8e23a8cced7d88a3a8be04bfd109ec8e6cab4da654c03814858c11030fa350"}
Oct 31 01:54:10.812516 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:10.812473    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:10.814293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:10.814260    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 01:54:10.814429 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:10.814308    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 01:54:11.813052 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:11.813012    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 01:54:11.813461 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:11.813073    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 01:54:12.811998 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:12.811964    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 01:54:12.812169 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:12.812014    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 01:54:13.812817 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:13.812777    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 01:54:13.813249 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:13.812845    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 01:54:14.818682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:14.818123    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 01:54:14.818682 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:14.818192    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 01:54:15.812593 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:15.812538    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 01:54:15.812864 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:15.812621    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 01:54:17.855838 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:17.855785    5718 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:17.856250 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:17.855879    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:18.203051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:18.202936    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:18.203352 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:18.203321    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:18.204318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:18.204294    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 01:54:18.334745 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:18.334694    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podStartSLOduration=10.33467885 podStartE2EDuration="10.33467885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:10.385818181 +0000 UTC m=+2966.548961608" watchObservedRunningTime="2026-10-31 01:54:18.33467885 +0000 UTC m=+2974.497822265"
Oct 31 01:54:20.676902 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:20.676861    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"]
Oct 31 01:54:20.677402 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:20.677124    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy" containerID="cri-o://1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1" gracePeriod=30
Oct 31 01:54:25.939557 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:25.939498    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:54:26.022820 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.022788    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.022836    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.022871    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.022915    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.022949    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.022966    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023045    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023104    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023141    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw\") pod \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\" (UID: \"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b\") "
Oct 31 01:54:26.023295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023158    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.023295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023191    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.023295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023200    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.023618 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023364    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.023618 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023542    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:26.023762 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023748    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.023821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023770    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.023821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023785    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.023821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023803    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.023821 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.023818    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.025904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.025874    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.025904 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.025887    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:26.026154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.026128    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "kube-api-access-qklpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.026154 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.026142    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token" pod "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" (UID: "4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.124776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.124742    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.124776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.124770    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklpw\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-kube-api-access-qklpw\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.124776 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.124781    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.125088 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.124790    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.289097 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.289065    5718 generic.go:356] "Generic (PLEG): container finished" podID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerID="1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1" exitCode=0
Oct 31 01:54:26.289293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.289110    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" event={"ID":"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b","Type":"ContainerDied","Data":"1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1"}
Oct 31 01:54:26.289293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.289114    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"
Oct 31 01:54:26.289293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.289152    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7" event={"ID":"4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b","Type":"ContainerDied","Data":"60230415af972480bbf5606d6be84aae3114bfec7d429be7e58c872e2242d1eb"}
Oct 31 01:54:26.289293 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.289175    5718 scope.go:122] "RemoveContainer" containerID="1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1"
Oct 31 01:54:26.299552 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.299527    5718 scope.go:122] "RemoveContainer" containerID="1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1"
Oct 31 01:54:26.299859 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:54:26.299840    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1\": container with ID starting with 1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1 not found: ID does not exist" containerID="1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1"
Oct 31 01:54:26.299923 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.299869    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1"} err="failed to get container status \"1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1\": rpc error: code = NotFound desc = could not find container \"1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1\": container with ID starting with 1fedacf9a9e37fe10634bf175aa27a59f0ff1877a17dbb32a88056c4bc94cbf1 not found: ID does not exist"
Oct 31 01:54:26.454892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.454852    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"]
Oct 31 01:54:26.463009 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.462977    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-hq4f7"]
Oct 31 01:54:26.485782 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.485745    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"]
Oct 31 01:54:26.486251 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.486238    5718 cpu_manager.go:414] "containerMap: removing container" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy"
Oct 31 01:54:26.486318 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.486252    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy"
Oct 31 01:54:26.486356 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.486344    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" containerName="istio-proxy"
Oct 31 01:54:26.491159 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.491137    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.577895 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.577810    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"]
Oct 31 01:54:26.630955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.630924    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631114 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.630975    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwkz\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631114 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631101    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631130    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631199 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631169    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631336 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631316    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631379 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631353    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631478 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631463    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.631514 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.631500    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732259    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732503 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732356    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732503 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732386    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732503 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732422    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732503 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732448    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732519    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732554    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwkz\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732593    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732615    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732784    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732826    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.732861 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732836    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.733305 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.732877    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.733447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.733423    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.735134 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.735114    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.735344 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.735326    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.793326 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.793296    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.793522 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.793499    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwkz\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:26.806390 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:26.806363    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:27.233076 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:27.233043    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"]
Oct 31 01:54:27.234302 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:54:27.234276    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246c83a2_0d9e_4460_ae0c_9601b2cfa293.slice/crio-49d9efbcc999423a48196cef820480e8851efa61e8a1712bba813b3e88299473 WatchSource:0}: Error finding container 49d9efbcc999423a48196cef820480e8851efa61e8a1712bba813b3e88299473: Status 404 returned error can't find the container with id 49d9efbcc999423a48196cef820480e8851efa61e8a1712bba813b3e88299473
Oct 31 01:54:27.236964 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:27.236935    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:54:27.237050 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:27.237005    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:54:27.237050 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:27.237034    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:54:27.295974 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:27.295943    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" event={"ID":"246c83a2-0d9e-4460-ae0c-9601b2cfa293","Type":"ContainerStarted","Data":"49d9efbcc999423a48196cef820480e8851efa61e8a1712bba813b3e88299473"}
Oct 31 01:54:27.976298 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:27.976260    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b" path="/var/lib/kubelet/pods/4feddf28-bfd9-4eba-8b81-6a56e8bbdc0b/volumes"
Oct 31 01:54:28.306681 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:28.306628    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" event={"ID":"246c83a2-0d9e-4460-ae0c-9601b2cfa293","Type":"ContainerStarted","Data":"1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907"}
Oct 31 01:54:28.519328 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:28.519047    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podStartSLOduration=2.519028858 podStartE2EDuration="2.519028858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:28.478975832 +0000 UTC m=+2984.642119251" watchObservedRunningTime="2026-10-31 01:54:28.519028858 +0000 UTC m=+2984.682172276"
Oct 31 01:54:28.812067 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:28.812031    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:28.813515 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:28.813488    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:54:28.813700 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:28.813537    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:54:29.807718 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:29.807644    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:54:29.808233 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:29.807754    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:54:30.806837 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:30.806796    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:54:30.807028 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:30.806857    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:54:31.806875 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:31.806838    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:54:31.807273 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:31.806902    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:54:32.806962 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:32.806923    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:54:32.807339 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:32.806999    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:54:33.807003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:33.806958    5718 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:54:33.807525 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:33.807030    5718 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 01:54:34.811160 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:34.811128    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:34.811588 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:34.811394    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:34.812264 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:54:34.812240    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 01:54:37.925598 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:54:37.925559    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:55:07.924520 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:55:07.924467    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:55:09.050182 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.050143    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"]
Oct 31 01:55:09.055773 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.055737    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.109011 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.108978    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:55:09.109011 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.108989    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:55:09.109226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.108991    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:55:09.109226 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.109058    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-85vx5\""
Oct 31 01:55:09.142558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.142513    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.142784 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.142577    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.142784 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.142745    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.142874 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.142843    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.142963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.142947    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d642g\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.143024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.143007    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.143156 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.143138    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.162287 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.162247    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"]
Oct 31 01:55:09.244609 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244568    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d642g\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.244824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244680    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.244824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244754    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.244824 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244796    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.245006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244831    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.245006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244882    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.245006 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.244931    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.245955 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.245921    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.247317 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.247287    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.247448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.247318    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.247448 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.247405    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.247558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.247476    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.303302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.303224    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.303302 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.303241    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d642g\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbd4m\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.370150 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.370117    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:09.798472 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.798445    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"]
Oct 31 01:55:09.799426 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:55:09.799400    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb553ac_5f6d_48f8_9496_5b6b22e92314.slice/crio-7f9643ec7b54d4275135071c1307dc36cf712b75253ddd796aa149658e8885eb WatchSource:0}: Error finding container 7f9643ec7b54d4275135071c1307dc36cf712b75253ddd796aa149658e8885eb: Status 404 returned error can't find the container with id 7f9643ec7b54d4275135071c1307dc36cf712b75253ddd796aa149658e8885eb
Oct 31 01:55:09.801983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.801950    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:55:09.802089 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:09.802027    5718 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:55:10.725835 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:10.725796    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" event={"ID":"2eb553ac-5f6d-48f8-9496-5b6b22e92314","Type":"ContainerStarted","Data":"b369f8a15a968a6de76826d9d3a38d985077e00e7f753a39daf946f01290d1cf"}
Oct 31 01:55:10.726321 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:10.725860    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" event={"ID":"2eb553ac-5f6d-48f8-9496-5b6b22e92314","Type":"ContainerStarted","Data":"7f9643ec7b54d4275135071c1307dc36cf712b75253ddd796aa149658e8885eb"}
Oct 31 01:55:10.726321 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:10.725983    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:10.728665 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:10.728610    5718 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lbd4m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:55:10.728795 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:10.728708    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:55:10.937712 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:10.937598    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" podStartSLOduration=1.9375790670000002 podStartE2EDuration="1.937579067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:10.896962582 +0000 UTC m=+3027.060105996" watchObservedRunningTime="2026-10-31 01:55:10.937579067 +0000 UTC m=+3027.100722480"
Oct 31 01:55:11.732909 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:55:11.732872    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 01:55:37.924587 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:55:37.924542    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:56:07.926322 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:56:07.926232    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:56:37.924104 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:56:37.924063    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:57:07.925524 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:57:07.925481    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:57:11.792963 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:11.792922    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"]
Oct 31 01:57:11.794191 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:11.794111    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" podUID="2a1f06cc-9f39-4004-b87b-2859da22601d" containerName="maas-api" containerID="cri-o://a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c" gracePeriod=30
Oct 31 01:57:12.105119 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.105091    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:57:12.151361 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.151329    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca\") pod \"2a1f06cc-9f39-4004-b87b-2859da22601d\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") "
Oct 31 01:57:12.151545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.151376    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j\" (UniqueName: \"kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j\") pod \"2a1f06cc-9f39-4004-b87b-2859da22601d\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") "
Oct 31 01:57:12.151545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.151466    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs\") pod \"2a1f06cc-9f39-4004-b87b-2859da22601d\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") "
Oct 31 01:57:12.151685 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.151570    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\") pod \"2a1f06cc-9f39-4004-b87b-2859da22601d\" (UID: \"2a1f06cc-9f39-4004-b87b-2859da22601d\") "
Oct 31 01:57:12.151828 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.151801    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca" pod "2a1f06cc-9f39-4004-b87b-2859da22601d" (UID: "2a1f06cc-9f39-4004-b87b-2859da22601d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:12.152253 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.152228    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1f06cc-9f39-4004-b87b-2859da22601d-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.154350 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.154322    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls" pod "2a1f06cc-9f39-4004-b87b-2859da22601d" (UID: "2a1f06cc-9f39-4004-b87b-2859da22601d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:12.154433 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.154345    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j" pod "2a1f06cc-9f39-4004-b87b-2859da22601d" (UID: "2a1f06cc-9f39-4004-b87b-2859da22601d"). InnerVolumeSpecName "kube-api-access-hcq6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:12.154474 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.154448    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs" pod "2a1f06cc-9f39-4004-b87b-2859da22601d" (UID: "2a1f06cc-9f39-4004-b87b-2859da22601d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:12.270455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.270417    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.270455 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.270454    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1f06cc-9f39-4004-b87b-2859da22601d-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.270704 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.270467    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcq6j\" (UniqueName: \"kubernetes.io/projected/2a1f06cc-9f39-4004-b87b-2859da22601d-kube-api-access-hcq6j\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:12.478086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.477987    5718 generic.go:356] "Generic (PLEG): container finished" podID="2a1f06cc-9f39-4004-b87b-2859da22601d" containerID="a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c" exitCode=0
Oct 31 01:57:12.478086 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.478042    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"
Oct 31 01:57:12.478307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.478079    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" event={"ID":"2a1f06cc-9f39-4004-b87b-2859da22601d","Type":"ContainerDied","Data":"a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c"}
Oct 31 01:57:12.478307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.478133    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q" event={"ID":"2a1f06cc-9f39-4004-b87b-2859da22601d","Type":"ContainerDied","Data":"2b9283475391d5123d0f14ee633f6eb398997654933f9e9121e8e411178c9085"}
Oct 31 01:57:12.478307 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.478155    5718 scope.go:122] "RemoveContainer" containerID="a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c"
Oct 31 01:57:12.489024 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.489004    5718 scope.go:122] "RemoveContainer" containerID="a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c"
Oct 31 01:57:12.489299 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:57:12.489281    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c\": container with ID starting with a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c not found: ID does not exist" containerID="a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c"
Oct 31 01:57:12.489363 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.489312    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c"} err="failed to get container status \"a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c\": rpc error: code = NotFound desc = could not find container \"a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c\": container with ID starting with a612f3284e5e490d8847e80d868cd3b20edf2d9c30e370347bfeb4600a56712c not found: ID does not exist"
Oct 31 01:57:12.649178 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.645322    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"]
Oct 31 01:57:12.649178 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:12.649121    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-cd0c98-6467c8f9ff-j755q"]
Oct 31 01:57:13.967691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:13.967641    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1f06cc-9f39-4004-b87b-2859da22601d" path="/var/lib/kubelet/pods/2a1f06cc-9f39-4004-b87b-2859da22601d/volumes"
Oct 31 01:57:20.804092 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:20.804057    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"]
Oct 31 01:57:20.804551 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:20.804274    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" containerID="cri-o://3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f" gracePeriod=30
Oct 31 01:57:25.729884 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:25.729799    5718 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:57:25.730370 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:25.730340    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:57:26.141405 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.141380    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:57:26.256800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.256701    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.256800 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.256778    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.256849    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.256903    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.256934    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257039 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.256966    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257050    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257087    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257132    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:26.257248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257132    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.257248 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257169    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy\") pod \"f30d5d30-3032-4f23-ae72-8993c39980de\" (UID: \"f30d5d30-3032-4f23-ae72-8993c39980de\") "
Oct 31 01:57:26.257486 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257338    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.257735 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257589    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.257880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257792    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.257880 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257824    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.258003 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.257785    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.258094 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.258076    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d5d30-3032-4f23-ae72-8993c39980de-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.258300 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.258256    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.259692 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.259668    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "kube-api-access-4stkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:26.259798 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.259753    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:57:26.259874 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.259859    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:26.259959 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.259934    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy" pod "f30d5d30-3032-4f23-ae72-8993c39980de" (UID: "f30d5d30-3032-4f23-ae72-8993c39980de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.359454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.359410    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.359454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.359441    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stkf\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-kube-api-access-4stkf\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.359454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.359451    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d5d30-3032-4f23-ae72-8993c39980de-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.359454 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.359462    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d5d30-3032-4f23-ae72-8993c39980de-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.359884 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.359470    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d5d30-3032-4f23-ae72-8993c39980de-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.577965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.577874    5718 generic.go:356] "Generic (PLEG): container finished" podID="f30d5d30-3032-4f23-ae72-8993c39980de" containerID="3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f" exitCode=0
Oct 31 01:57:26.577965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.577928    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"
Oct 31 01:57:26.577965 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.577950    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" event={"ID":"f30d5d30-3032-4f23-ae72-8993c39980de","Type":"ContainerDied","Data":"3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f"}
Oct 31 01:57:26.578220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.578010    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz" event={"ID":"f30d5d30-3032-4f23-ae72-8993c39980de","Type":"ContainerDied","Data":"a64da2613949d991209a6f7336f4d648f89b9102380fcaaf1f5a913b4ee15efa"}
Oct 31 01:57:26.578220 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.578032    5718 scope.go:122] "RemoveContainer" containerID="3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f"
Oct 31 01:57:26.596795 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.594792    5718 scope.go:122] "RemoveContainer" containerID="3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f"
Oct 31 01:57:26.596795 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:57:26.596708    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f\": container with ID starting with 3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f not found: ID does not exist" containerID="3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f"
Oct 31 01:57:26.596795 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.596750    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f"} err="failed to get container status \"3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f\": rpc error: code = NotFound desc = could not find container \"3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f\": container with ID starting with 3cd156cc975909ea6c6e45b4c5698377fb80c7ac7ce99478c1cd22f2a0947b2f not found: ID does not exist"
Oct 31 01:57:26.744257 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.744218    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"]
Oct 31 01:57:26.796403 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:26.796367    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-5hdbz"]
Oct 31 01:57:27.967348 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:57:27.967318    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" path="/var/lib/kubelet/pods/f30d5d30-3032-4f23-ae72-8993c39980de/volumes"
Oct 31 01:57:37.923098 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:57:37.923055    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:58:05.971295 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971264    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"]
Oct 31 01:58:05.971841 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971798    5718 cpu_manager.go:414] "containerMap: removing container" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy"
Oct 31 01:58:05.971841 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971815    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy"
Oct 31 01:58:05.971953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971844    5718 cpu_manager.go:414] "containerMap: removing container" podUID="2a1f06cc-9f39-4004-b87b-2859da22601d" containerName="maas-api"
Oct 31 01:58:05.971953 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971852    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1f06cc-9f39-4004-b87b-2859da22601d" containerName="maas-api"
Oct 31 01:58:05.972035 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971978    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30d5d30-3032-4f23-ae72-8993c39980de" containerName="istio-proxy"
Oct 31 01:58:05.972035 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.971994    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1f06cc-9f39-4004-b87b-2859da22601d" containerName="maas-api"
Oct 31 01:58:05.976892 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:05.976869    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.030939 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.030907    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"]
Oct 31 01:58:06.078264 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.078224    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-98c748-serving-cert\""
Oct 31 01:58:06.091084 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.091042    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895h4\" (UniqueName: \"kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.091294 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.091207    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.091399 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.091379    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.091812 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.091766    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.192724 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.192642    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.192917 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.192786    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895h4\" (UniqueName: \"kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.192917 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.192893    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.193043 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.193019    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.193488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.193455    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.195801 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.195779    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.195913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.195852    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.254431 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.254398    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895h4\" (UniqueName: \"kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4\") pod \"maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.290769 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.290698    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:06.670488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.670456    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"]
Oct 31 01:58:06.672131 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:58:06.672104    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145e629f_8852_4a5e_9286_5ad9e17eb96e.slice/crio-2490472ade3f4d79207e9c46f68f2406483e3af311c3e0d6378fe95666505f35 WatchSource:0}: Error finding container 2490472ade3f4d79207e9c46f68f2406483e3af311c3e0d6378fe95666505f35: Status 404 returned error can't find the container with id 2490472ade3f4d79207e9c46f68f2406483e3af311c3e0d6378fe95666505f35
Oct 31 01:58:06.673447 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.673429    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:58:06.822708 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:06.822645    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr" event={"ID":"145e629f-8852-4a5e-9286-5ad9e17eb96e","Type":"ContainerStarted","Data":"2490472ade3f4d79207e9c46f68f2406483e3af311c3e0d6378fe95666505f35"}
Oct 31 01:58:07.830561 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:07.830522    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr" event={"ID":"145e629f-8852-4a5e-9286-5ad9e17eb96e","Type":"ContainerStarted","Data":"0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6"}
Oct 31 01:58:07.830994 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:07.830717    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:07.924531 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:58:07.924451    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:58:07.989936 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:07.989866    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr" podStartSLOduration=2.054827716 podStartE2EDuration="2.989848243s" totalImagesPullingTime="935.020527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:07.947900737 +0000 UTC m=+3204.111044157" watchObservedRunningTime="2026-10-31 01:58:07.989848243 +0000 UTC m=+3204.152991661"
Oct 31 01:58:13.841721 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:13.841691    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 01:58:34.655282 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.655245    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"]
Oct 31 01:58:34.659214 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.659190    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.743913 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.743883    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.744081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.743932    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmsj\" (UniqueName: \"kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.744081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.744034    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.744081 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.744063    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.765828 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.765791    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c4d438-serving-cert\""
Oct 31 01:58:34.768345 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.768315    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"]
Oct 31 01:58:34.845851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.845816    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.845851 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.845858    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmsj\" (UniqueName: \"kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.846068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.845933    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.846068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.845986    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.846068 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:58:34.846023    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-serving-cert: secret "maas-api-e2e-shared-b-w3-c4d438-serving-cert" not found
Oct 31 01:58:34.846168 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:58:34.846075    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls podName:b91d1b08-9e21-4a52-a31d-9c7989db39b6 nodeName:}" failed. No retries permitted until 2026-10-31 01:58:35.346058068 +0000 UTC m=+3231.509201466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" (UID: "b91d1b08-9e21-4a52-a31d-9c7989db39b6") : secret "maas-api-e2e-shared-b-w3-c4d438-serving-cert" not found
Oct 31 01:58:34.846581 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.846553    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.848488 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.848466    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:34.906386 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:34.906316    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmsj\" (UniqueName: \"kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:35.353099 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:35.353057    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:35.355685 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:35.355645    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:35.572897 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:35.572860    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:35.953640 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:35.953618    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"]
Oct 31 01:58:35.955632 ip-10-0-141-86 kubenswrapper[5718]: W1031 01:58:35.955595    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91d1b08_9e21_4a52_a31d_9c7989db39b6.slice/crio-7dd8d09844f1f4926ebc82ba07018508f86077ce0a7abeb2a86bdbee94803311 WatchSource:0}: Error finding container 7dd8d09844f1f4926ebc82ba07018508f86077ce0a7abeb2a86bdbee94803311: Status 404 returned error can't find the container with id 7dd8d09844f1f4926ebc82ba07018508f86077ce0a7abeb2a86bdbee94803311
Oct 31 01:58:36.000523 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:36.000486    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" event={"ID":"b91d1b08-9e21-4a52-a31d-9c7989db39b6","Type":"ContainerStarted","Data":"7dd8d09844f1f4926ebc82ba07018508f86077ce0a7abeb2a86bdbee94803311"}
Oct 31 01:58:37.008373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:37.008331    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" event={"ID":"b91d1b08-9e21-4a52-a31d-9c7989db39b6","Type":"ContainerStarted","Data":"b70ab85d0308d84addc017cd00d1f9da80e054ab8148f766058d0e9fa147e257"}
Oct 31 01:58:37.008865 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:37.008447    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:58:37.151068 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:37.150960    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" podStartSLOduration=2.267114951 podStartE2EDuration="3.150943658s" totalImagesPullingTime="883.828707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:37.14917605 +0000 UTC m=+3233.312319468" watchObservedRunningTime="2026-10-31 01:58:37.150943658 +0000 UTC m=+3233.314087168"
Oct 31 01:58:37.925149 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:58:37.925112    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:58:43.021729 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:58:43.021695    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 01:59:07.925154 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:59:07.925067    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 01:59:25.876384 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:25.876345    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"]
Oct 31 01:59:25.876939 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:25.876545    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy" containerID="cri-o://5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e" gracePeriod=30
Oct 31 01:59:31.137915 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.137889    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:59:31.242741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242619    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.242741 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242719    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.242983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242793    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.242983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242824    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.242983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242869    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.242983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242918    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.242983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242926    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:31.242983 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242951    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.243274 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.242991    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.243274 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243061    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert\") pod \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\" (UID: \"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d\") "
Oct 31 01:59:31.243370 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243339    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:31.243558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243437    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:31.243558 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243542    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:31.243751 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243671    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:31.243961 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243939    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.244020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243968    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.244020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.243986    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.244020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.244002    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.244020 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.244016    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.245243 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.245219    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:31.245545 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.245524    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "kube-api-access-pv4t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:31.245701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.245679    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:59:31.245701 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.245686    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy" pod "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" (UID: "1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:31.329832 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.329791    5718 generic.go:356] "Generic (PLEG): container finished" podID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerID="5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e" exitCode=0
Oct 31 01:59:31.330051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.329848    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"
Oct 31 01:59:31.330051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.329891    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" event={"ID":"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d","Type":"ContainerDied","Data":"5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e"}
Oct 31 01:59:31.330051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.329948    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx" event={"ID":"1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d","Type":"ContainerDied","Data":"6fffe3de4365db76161c7d976f32406912513a78cc532852a37220ec6a0a40c2"}
Oct 31 01:59:31.330051 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.329975    5718 scope.go:122] "RemoveContainer" containerID="5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e"
Oct 31 01:59:31.342465 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.342441    5718 scope.go:122] "RemoveContainer" containerID="5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e"
Oct 31 01:59:31.342810 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:59:31.342785    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e\": container with ID starting with 5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e not found: ID does not exist" containerID="5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e"
Oct 31 01:59:31.342937 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.342820    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e"} err="failed to get container status \"5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e\": rpc error: code = NotFound desc = could not find container \"5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e\": container with ID starting with 5ee93c7286dac7fafb9fd52db7cfe4dfec379225f5c69c7210822602dcdfa66e not found: ID does not exist"
Oct 31 01:59:31.345507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.345484    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.345507 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.345506    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.345691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.345517    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4t5\" (UniqueName: \"kubernetes.io/projected/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-kube-api-access-pv4t5\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.345691 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.345530    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:31.496362 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.496325    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"]
Oct 31 01:59:31.501373 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.501340    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-c2vpx"]
Oct 31 01:59:31.975071 ip-10-0-141-86 kubenswrapper[5718]: I1031 01:59:31.974985    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" path="/var/lib/kubelet/pods/1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d/volumes"
Oct 31 01:59:37.926114 ip-10-0-141-86 kubenswrapper[5718]: E1031 01:59:37.926076    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:00:00.159776 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.159741    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"]
Oct 31 02:00:00.160277 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.160244    5718 cpu_manager.go:414] "containerMap: removing container" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy"
Oct 31 02:00:00.160277 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.160256    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy"
Oct 31 02:00:00.160387 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.160375    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0a5d9c-4c28-4951-969f-1c6b7cf99b7d" containerName="istio-proxy"
Oct 31 02:00:00.163891 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.163874    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.188030 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.188001    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"]
Oct 31 02:00:00.191907 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.191878    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"]
Oct 31 02:00:00.192041 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.191989    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.195403 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.195381    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.210325 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.210297    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"]
Oct 31 02:00:00.214421 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.214402    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"]
Oct 31 02:00:00.214580 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.214509    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.217912 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.217892    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.236301 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.236279    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:00:00.265233 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.265202    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"]
Oct 31 02:00:00.307466 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307427    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9t5\" (UniqueName: \"kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.307685 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307558    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.307685 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307624    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.307913 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307700    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw87\" (UniqueName: \"kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.307913 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307746    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdnt\" (UniqueName: \"kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.307913 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307779    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.307913 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307905    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tbzs\" (UniqueName: \"kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.308205 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.307970    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.308205 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.308003    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.308903 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.308284    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2fc\" (UniqueName: \"kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.315224 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.315195    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"]
Oct 31 02:00:00.318155 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.318129    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"]
Oct 31 02:00:00.320321 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.320294    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"]
Oct 31 02:00:00.378623 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.378592    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"]
Oct 31 02:00:00.410451 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410352    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.410451 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410422    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw87\" (UniqueName: \"kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.410719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410465    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdnt\" (UniqueName: \"kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.410719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410500    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.410719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410562    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tbzs\" (UniqueName: \"kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.410719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410625    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.410719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410681    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.411232 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410776    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2fc\" (UniqueName: \"kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.411232 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410841    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9t5\" (UniqueName: \"kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.411232 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.410894    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.411232 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.411191    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.411606 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.411571    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.411726 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.411607    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.411990 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.411941    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.412289 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.412245    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.468369 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.468338    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw87\" (UniqueName: \"kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.468529 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.468388    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2fc\" (UniqueName: \"kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.468529 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.468467    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9t5\" (UniqueName: \"kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.468609 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.468530    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tbzs\" (UniqueName: \"kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.468609 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.468536    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdnt\" (UniqueName: \"kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.477339 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.477317    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:00.507184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.507158    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:00.514382 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.514355    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:00.529974 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.528541    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:00.536121 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.534728    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:00.859178 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.859144    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"]
Oct 31 02:00:00.861086 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:00:00.861059    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a677ad8_7a45_4180_9075_0ae5caa8b566.slice/crio-02832ba3330680f3ba418d0aadfb95731c1afb45daca12b5b481efa6948e3679 WatchSource:0}: Error finding container 02832ba3330680f3ba418d0aadfb95731c1afb45daca12b5b481efa6948e3679: Status 404 returned error can't find the container with id 02832ba3330680f3ba418d0aadfb95731c1afb45daca12b5b481efa6948e3679
Oct 31 02:00:00.912524 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.912496    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"]
Oct 31 02:00:00.913629 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:00:00.913602    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8a2102_f8b9_4e38_96bb_48e179759fe8.slice/crio-e3d870d9848cdd29c7a532a2741a17ffdb2b8da9a6fcdd753892813b0fbd1195 WatchSource:0}: Error finding container e3d870d9848cdd29c7a532a2741a17ffdb2b8da9a6fcdd753892813b0fbd1195: Status 404 returned error can't find the container with id e3d870d9848cdd29c7a532a2741a17ffdb2b8da9a6fcdd753892813b0fbd1195
Oct 31 02:00:00.959806 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.959774    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"]
Oct 31 02:00:00.960776 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:00:00.960734    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f44f25a_f1ab_405a_99ce_1e38f335bd83.slice/crio-0efa096b404a3d23c4ced297ea4f3b7c1d433259381cd4d14a45ce13a89cf93b WatchSource:0}: Error finding container 0efa096b404a3d23c4ced297ea4f3b7c1d433259381cd4d14a45ce13a89cf93b: Status 404 returned error can't find the container with id 0efa096b404a3d23c4ced297ea4f3b7c1d433259381cd4d14a45ce13a89cf93b
Oct 31 02:00:00.973438 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.973408    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"]
Oct 31 02:00:00.976192 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:00:00.976166    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d18f623_68e0_41c6_92d8_29e50b8987ab.slice/crio-b079b1256915936a78b2c286a1ce16a262a941b7651bc8f7325decd2c8c1e9d9 WatchSource:0}: Error finding container b079b1256915936a78b2c286a1ce16a262a941b7651bc8f7325decd2c8c1e9d9: Status 404 returned error can't find the container with id b079b1256915936a78b2c286a1ce16a262a941b7651bc8f7325decd2c8c1e9d9
Oct 31 02:00:00.976691 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:00.976647    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"]
Oct 31 02:00:00.977930 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:00:00.977905    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ef7ecc_0532_47b7_8b49_b272d506f569.slice/crio-451ce77ed9bb7d5c25a90dc32de88f3cd6cb75c206d40151058a383a3e0ae6a6 WatchSource:0}: Error finding container 451ce77ed9bb7d5c25a90dc32de88f3cd6cb75c206d40151058a383a3e0ae6a6: Status 404 returned error can't find the container with id 451ce77ed9bb7d5c25a90dc32de88f3cd6cb75c206d40151058a383a3e0ae6a6
Oct 31 02:00:01.517158 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.517041    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerStarted","Data":"ee496312f349ed5222b692342bc1faf6b7b74a9b260366ccde4d406c1a59ac09"}
Oct 31 02:00:01.517158 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.517153    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerStarted","Data":"e3d870d9848cdd29c7a532a2741a17ffdb2b8da9a6fcdd753892813b0fbd1195"}
Oct 31 02:00:01.518696 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.518671    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerStarted","Data":"5bf4066822b5109270b5d25e6b3695a9ab08dc87a00dcd82b302c553521a277f"}
Oct 31 02:00:01.518829 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.518710    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerStarted","Data":"02832ba3330680f3ba418d0aadfb95731c1afb45daca12b5b481efa6948e3679"}
Oct 31 02:00:01.520269 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.520245    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerStarted","Data":"6a58d5ed78d5f3b8cb7866da97d5a2ed7a74b138863cb8a222591fb7057409c8"}
Oct 31 02:00:01.520366 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.520295    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerStarted","Data":"451ce77ed9bb7d5c25a90dc32de88f3cd6cb75c206d40151058a383a3e0ae6a6"}
Oct 31 02:00:01.521877 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.521852    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerStarted","Data":"352be91be3485639b0724ef30de1822ef8b1b89999b588ccf361cfb8ee041e5d"}
Oct 31 02:00:01.521968 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.521900    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerStarted","Data":"b079b1256915936a78b2c286a1ce16a262a941b7651bc8f7325decd2c8c1e9d9"}
Oct 31 02:00:01.523256 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.523232    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerStarted","Data":"23d66badd8fb92af7d699499e6f1407e7f3060898a0b57964dfd83f513a5109f"}
Oct 31 02:00:01.523350 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.523269    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerStarted","Data":"0efa096b404a3d23c4ced297ea4f3b7c1d433259381cd4d14a45ce13a89cf93b"}
Oct 31 02:00:01.632452 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.632404    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" podStartSLOduration=1.632390628 podStartE2EDuration="1.632390628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.632152871 +0000 UTC m=+3317.795296286" watchObservedRunningTime="2026-10-31 02:00:01.632390628 +0000 UTC m=+3317.795534044"
Oct 31 02:00:01.746695 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.745931    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" podStartSLOduration=1.745911005 podStartE2EDuration="1.745911005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.745163989 +0000 UTC m=+3317.908307416" watchObservedRunningTime="2026-10-31 02:00:01.745911005 +0000 UTC m=+3317.909054421"
Oct 31 02:00:01.858953 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:01.858904    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" podStartSLOduration=1.858890128 podStartE2EDuration="1.858890128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.858447754 +0000 UTC m=+3318.021591171" watchObservedRunningTime="2026-10-31 02:00:01.858890128 +0000 UTC m=+3318.022033542"
Oct 31 02:00:02.090485 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.090433    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" podStartSLOduration=2.090417872 podStartE2EDuration="2.090417872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.088901439 +0000 UTC m=+3318.252044846" watchObservedRunningTime="2026-10-31 02:00:02.090417872 +0000 UTC m=+3318.253561286"
Oct 31 02:00:02.091566 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.091508    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" podStartSLOduration=2.091495643 podStartE2EDuration="2.091495643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.975451319 +0000 UTC m=+3318.138594734" watchObservedRunningTime="2026-10-31 02:00:02.091495643 +0000 UTC m=+3318.254639058"
Oct 31 02:00:02.534370 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.534335    5718 generic.go:356] "Generic (PLEG): container finished" podID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerID="6a58d5ed78d5f3b8cb7866da97d5a2ed7a74b138863cb8a222591fb7057409c8" exitCode=60
Oct 31 02:00:02.534934 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.534434    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerDied","Data":"6a58d5ed78d5f3b8cb7866da97d5a2ed7a74b138863cb8a222591fb7057409c8"}
Oct 31 02:00:02.534934 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.534786    5718 scope.go:122] "RemoveContainer" containerID="6a58d5ed78d5f3b8cb7866da97d5a2ed7a74b138863cb8a222591fb7057409c8"
Oct 31 02:00:02.536122 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.536101    5718 generic.go:356] "Generic (PLEG): container finished" podID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerID="352be91be3485639b0724ef30de1822ef8b1b89999b588ccf361cfb8ee041e5d" exitCode=60
Oct 31 02:00:02.536229 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.536174    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerDied","Data":"352be91be3485639b0724ef30de1822ef8b1b89999b588ccf361cfb8ee041e5d"}
Oct 31 02:00:02.536516 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.536429    5718 scope.go:122] "RemoveContainer" containerID="352be91be3485639b0724ef30de1822ef8b1b89999b588ccf361cfb8ee041e5d"
Oct 31 02:00:02.538136 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.537986    5718 generic.go:356] "Generic (PLEG): container finished" podID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerID="23d66badd8fb92af7d699499e6f1407e7f3060898a0b57964dfd83f513a5109f" exitCode=60
Oct 31 02:00:02.538136 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.538104    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerDied","Data":"23d66badd8fb92af7d699499e6f1407e7f3060898a0b57964dfd83f513a5109f"}
Oct 31 02:00:02.538305 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.538281    5718 scope.go:122] "RemoveContainer" containerID="23d66badd8fb92af7d699499e6f1407e7f3060898a0b57964dfd83f513a5109f"
Oct 31 02:00:02.539813 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.539792    5718 generic.go:356] "Generic (PLEG): container finished" podID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerID="ee496312f349ed5222b692342bc1faf6b7b74a9b260366ccde4d406c1a59ac09" exitCode=60
Oct 31 02:00:02.539916 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.539824    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerDied","Data":"ee496312f349ed5222b692342bc1faf6b7b74a9b260366ccde4d406c1a59ac09"}
Oct 31 02:00:02.540073 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.540055    5718 scope.go:122] "RemoveContainer" containerID="ee496312f349ed5222b692342bc1faf6b7b74a9b260366ccde4d406c1a59ac09"
Oct 31 02:00:02.541473 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.541453    5718 generic.go:356] "Generic (PLEG): container finished" podID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerID="5bf4066822b5109270b5d25e6b3695a9ab08dc87a00dcd82b302c553521a277f" exitCode=60
Oct 31 02:00:02.541584 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.541487    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerDied","Data":"5bf4066822b5109270b5d25e6b3695a9ab08dc87a00dcd82b302c553521a277f"}
Oct 31 02:00:02.541727 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:02.541712    5718 scope.go:122] "RemoveContainer" containerID="5bf4066822b5109270b5d25e6b3695a9ab08dc87a00dcd82b302c553521a277f"
Oct 31 02:00:03.550953 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.550922    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerStarted","Data":"3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a"}
Oct 31 02:00:03.552972 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.552945    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerStarted","Data":"48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976"}
Oct 31 02:00:03.554941 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.554917    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerStarted","Data":"0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2"}
Oct 31 02:00:03.557488 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.557460    5718 generic.go:356] "Generic (PLEG): container finished" podID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerID="b99efc09b993d2a896d1161cb6a2c447a36f62a41a8b914d1f69566d2e246d31" exitCode=60
Oct 31 02:00:03.557583 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.557552    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerDied","Data":"b99efc09b993d2a896d1161cb6a2c447a36f62a41a8b914d1f69566d2e246d31"}
Oct 31 02:00:03.557691 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.557600    5718 scope.go:122] "RemoveContainer" containerID="352be91be3485639b0724ef30de1822ef8b1b89999b588ccf361cfb8ee041e5d"
Oct 31 02:00:03.557935 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.557913    5718 scope.go:122] "RemoveContainer" containerID="b99efc09b993d2a896d1161cb6a2c447a36f62a41a8b914d1f69566d2e246d31"
Oct 31 02:00:03.558150 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:03.558133    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz_odh-ai-gateway-infra(8d18f623-68e0-41c6-92d8-29e50b8987ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab"
Oct 31 02:00:03.560454 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:03.560434    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerStarted","Data":"85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f"}
Oct 31 02:00:04.572316 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.572271    5718 generic.go:356] "Generic (PLEG): container finished" podID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerID="48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976" exitCode=60
Oct 31 02:00:04.572819 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.572407    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerDied","Data":"48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976"}
Oct 31 02:00:04.572819 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.572484    5718 scope.go:122] "RemoveContainer" containerID="5bf4066822b5109270b5d25e6b3695a9ab08dc87a00dcd82b302c553521a277f"
Oct 31 02:00:04.572819 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.572741    5718 scope.go:122] "RemoveContainer" containerID="48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976"
Oct 31 02:00:04.573101 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:04.573078    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm_odh-ai-gateway-infra(6a677ad8-7a45-4180-9075-0ae5caa8b566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566"
Oct 31 02:00:04.575226 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.575197    5718 generic.go:356] "Generic (PLEG): container finished" podID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerID="0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2" exitCode=60
Oct 31 02:00:04.575335 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.575281    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerDied","Data":"0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2"}
Oct 31 02:00:04.575521 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.575504    5718 scope.go:122] "RemoveContainer" containerID="0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2"
Oct 31 02:00:04.575797 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:04.575776    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj_odh-ai-gateway-infra(94ef7ecc-0532-47b7-8b49-b272d506f569)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569"
Oct 31 02:00:04.577830 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.577808    5718 scope.go:122] "RemoveContainer" containerID="b99efc09b993d2a896d1161cb6a2c447a36f62a41a8b914d1f69566d2e246d31"
Oct 31 02:00:04.578109 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:04.578085    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz_odh-ai-gateway-infra(8d18f623-68e0-41c6-92d8-29e50b8987ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab"
Oct 31 02:00:04.579705 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.579642    5718 generic.go:356] "Generic (PLEG): container finished" podID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerID="85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f" exitCode=60
Oct 31 02:00:04.579997 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.579968    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerDied","Data":"85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f"}
Oct 31 02:00:04.580119 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.580088    5718 scope.go:122] "RemoveContainer" containerID="85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f"
Oct 31 02:00:04.580366 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:04.580342    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht_odh-ai-gateway-infra(7f44f25a-f1ab-405a-99ce-1e38f335bd83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83"
Oct 31 02:00:04.582172 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.582149    5718 generic.go:356] "Generic (PLEG): container finished" podID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerID="3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a" exitCode=60
Oct 31 02:00:04.582327 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.582299    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerDied","Data":"3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a"}
Oct 31 02:00:04.582555 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.582539    5718 scope.go:122] "RemoveContainer" containerID="3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a"
Oct 31 02:00:04.582862 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:04.582838    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s_odh-ai-gateway-infra(ff8a2102-f8b9-4e38-96bb-48e179759fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8"
Oct 31 02:00:04.588509 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.588485    5718 scope.go:122] "RemoveContainer" containerID="6a58d5ed78d5f3b8cb7866da97d5a2ed7a74b138863cb8a222591fb7057409c8"
Oct 31 02:00:04.601773 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.601749    5718 scope.go:122] "RemoveContainer" containerID="23d66badd8fb92af7d699499e6f1407e7f3060898a0b57964dfd83f513a5109f"
Oct 31 02:00:04.615360 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:04.615330    5718 scope.go:122] "RemoveContainer" containerID="ee496312f349ed5222b692342bc1faf6b7b74a9b260366ccde4d406c1a59ac09"
Oct 31 02:00:05.590029 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:05.589996    5718 scope.go:122] "RemoveContainer" containerID="85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f"
Oct 31 02:00:05.590534 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:05.590351    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht_odh-ai-gateway-infra(7f44f25a-f1ab-405a-99ce-1e38f335bd83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83"
Oct 31 02:00:05.591753 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:05.591725    5718 scope.go:122] "RemoveContainer" containerID="3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a"
Oct 31 02:00:05.592040 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:05.592019    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s_odh-ai-gateway-infra(ff8a2102-f8b9-4e38-96bb-48e179759fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8"
Oct 31 02:00:05.593273 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:05.593254    5718 scope.go:122] "RemoveContainer" containerID="48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976"
Oct 31 02:00:05.593483 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:05.593455    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm_odh-ai-gateway-infra(6a677ad8-7a45-4180-9075-0ae5caa8b566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566"
Oct 31 02:00:05.594783 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:05.594767    5718 scope.go:122] "RemoveContainer" containerID="0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2"
Oct 31 02:00:05.594961 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:05.594944    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj_odh-ai-gateway-infra(94ef7ecc-0532-47b7-8b49-b272d506f569)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569"
Oct 31 02:00:07.924499 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:07.924455    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:00:14.960212 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:14.960178    5718 scope.go:122] "RemoveContainer" containerID="48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976"
Oct 31 02:00:14.962459 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:14.960286    5718 scope.go:122] "RemoveContainer" containerID="3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a"
Oct 31 02:00:15.658631 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:15.658538    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerStarted","Data":"713ab4b849f25d0f45023ba29227dbdf9b4f6bb1e02e6c191ead8e8cf34e41cc"}
Oct 31 02:00:15.660591 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:15.660563    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerStarted","Data":"8dcb63eb33e27685e9874e3be9a2a518ad42a19dbf2b5a1176234458f6bfecd0"}
Oct 31 02:00:16.670116 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.670078    5718 generic.go:356] "Generic (PLEG): container finished" podID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerID="713ab4b849f25d0f45023ba29227dbdf9b4f6bb1e02e6c191ead8e8cf34e41cc" exitCode=60
Oct 31 02:00:16.670589 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.670151    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerDied","Data":"713ab4b849f25d0f45023ba29227dbdf9b4f6bb1e02e6c191ead8e8cf34e41cc"}
Oct 31 02:00:16.670589 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.670230    5718 scope.go:122] "RemoveContainer" containerID="3ed7f63f7021359c9439c6013436926cb6f1ba5b4c98d101895ddcdee5c46e2a"
Oct 31 02:00:16.670768 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.670590    5718 scope.go:122] "RemoveContainer" containerID="713ab4b849f25d0f45023ba29227dbdf9b4f6bb1e02e6c191ead8e8cf34e41cc"
Oct 31 02:00:16.670967 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:16.670942    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s_odh-ai-gateway-infra(ff8a2102-f8b9-4e38-96bb-48e179759fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8"
Oct 31 02:00:16.673027 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.673006    5718 generic.go:356] "Generic (PLEG): container finished" podID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerID="8dcb63eb33e27685e9874e3be9a2a518ad42a19dbf2b5a1176234458f6bfecd0" exitCode=60
Oct 31 02:00:16.673118 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.673047    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerDied","Data":"8dcb63eb33e27685e9874e3be9a2a518ad42a19dbf2b5a1176234458f6bfecd0"}
Oct 31 02:00:16.673362 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.673346    5718 scope.go:122] "RemoveContainer" containerID="8dcb63eb33e27685e9874e3be9a2a518ad42a19dbf2b5a1176234458f6bfecd0"
Oct 31 02:00:16.673627 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:16.673609    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm_odh-ai-gateway-infra(6a677ad8-7a45-4180-9075-0ae5caa8b566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566"
Oct 31 02:00:16.682541 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.682525    5718 scope.go:122] "RemoveContainer" containerID="48378866df5bc49fd8775e06ca7054762ae00880a0ff13461865dab34e76b976"
Oct 31 02:00:16.798336 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.798296    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"]
Oct 31 02:00:16.902747 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.902704    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"]
Oct 31 02:00:16.960876 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:16.960777    5718 scope.go:122] "RemoveContainer" containerID="85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f"
Oct 31 02:00:17.681729 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:17.681622    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerStarted","Data":"a2c84308b44bb2181ca77978698779e2826e47a21caccd0d0e0ca6c06d6849c9"}
Oct 31 02:00:17.927425 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:17.927400    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:17.931046 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:17.931009    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:17.972588 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:17.972507    5718 scope.go:122] "RemoveContainer" containerID="0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2"
Oct 31 02:00:17.972793 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:17.972773    5718 scope.go:122] "RemoveContainer" containerID="b99efc09b993d2a896d1161cb6a2c447a36f62a41a8b914d1f69566d2e246d31"
Oct 31 02:00:18.038279 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.038254    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt\" (UniqueName: \"kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt\") pod \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") "
Oct 31 02:00:18.038421 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.038310    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc\" (UniqueName: \"kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc\") pod \"6a677ad8-7a45-4180-9075-0ae5caa8b566\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") "
Oct 31 02:00:18.038546 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.038529    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca\") pod \"6a677ad8-7a45-4180-9075-0ae5caa8b566\" (UID: \"6a677ad8-7a45-4180-9075-0ae5caa8b566\") "
Oct 31 02:00:18.038583 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.038572    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca\") pod \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\" (UID: \"ff8a2102-f8b9-4e38-96bb-48e179759fe8\") "
Oct 31 02:00:18.038933 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.038905    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca" pod "ff8a2102-f8b9-4e38-96bb-48e179759fe8" (UID: "ff8a2102-f8b9-4e38-96bb-48e179759fe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:18.038933 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.038918    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca" pod "6a677ad8-7a45-4180-9075-0ae5caa8b566" (UID: "6a677ad8-7a45-4180-9075-0ae5caa8b566"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:18.039280 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.039255    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a677ad8-7a45-4180-9075-0ae5caa8b566-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.039381 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.039286    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8a2102-f8b9-4e38-96bb-48e179759fe8-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.041157 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.041133    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt" pod "ff8a2102-f8b9-4e38-96bb-48e179759fe8" (UID: "ff8a2102-f8b9-4e38-96bb-48e179759fe8"). InnerVolumeSpecName "kube-api-access-7pdnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:18.041251 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.041155    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc" pod "6a677ad8-7a45-4180-9075-0ae5caa8b566" (UID: "6a677ad8-7a45-4180-9075-0ae5caa8b566"). InnerVolumeSpecName "kube-api-access-bw2fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:18.140825 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.140753    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdnt\" (UniqueName: \"kubernetes.io/projected/ff8a2102-f8b9-4e38-96bb-48e179759fe8-kube-api-access-7pdnt\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.140825 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.140791    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2fc\" (UniqueName: \"kubernetes.io/projected/6a677ad8-7a45-4180-9075-0ae5caa8b566-kube-api-access-bw2fc\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.694221 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.694139    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"
Oct 31 02:00:18.694221 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.694166    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s" event={"ID":"ff8a2102-f8b9-4e38-96bb-48e179759fe8","Type":"ContainerDied","Data":"e3d870d9848cdd29c7a532a2741a17ffdb2b8da9a6fcdd753892813b0fbd1195"}
Oct 31 02:00:18.694760 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.694232    5718 scope.go:122] "RemoveContainer" containerID="713ab4b849f25d0f45023ba29227dbdf9b4f6bb1e02e6c191ead8e8cf34e41cc"
Oct 31 02:00:18.695741 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.695719    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"
Oct 31 02:00:18.695878 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.695720    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm" event={"ID":"6a677ad8-7a45-4180-9075-0ae5caa8b566","Type":"ContainerDied","Data":"02832ba3330680f3ba418d0aadfb95731c1afb45daca12b5b481efa6948e3679"}
Oct 31 02:00:18.698074 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.698046    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerStarted","Data":"6750a8558576cf8ea82589161bfacd2e6cbca32e5abafec0674bf5fbdedd500d"}
Oct 31 02:00:18.700338 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.700057    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerStarted","Data":"96847b4c4d349819c66611c34b8155365ecad10ad756f449732e5b11eec31856"}
Oct 31 02:00:18.706445 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.706423    5718 scope.go:122] "RemoveContainer" containerID="8dcb63eb33e27685e9874e3be9a2a518ad42a19dbf2b5a1176234458f6bfecd0"
Oct 31 02:00:18.710566 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.710533    5718 generic.go:356] "Generic (PLEG): container finished" podID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerID="a2c84308b44bb2181ca77978698779e2826e47a21caccd0d0e0ca6c06d6849c9" exitCode=60
Oct 31 02:00:18.710638 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.710599    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerDied","Data":"a2c84308b44bb2181ca77978698779e2826e47a21caccd0d0e0ca6c06d6849c9"}
Oct 31 02:00:18.711239 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.711147    5718 scope.go:122] "RemoveContainer" containerID="a2c84308b44bb2181ca77978698779e2826e47a21caccd0d0e0ca6c06d6849c9"
Oct 31 02:00:18.711733 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:18.711679    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht_odh-ai-gateway-infra(7f44f25a-f1ab-405a-99ce-1e38f335bd83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83"
Oct 31 02:00:18.723788 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.723764    5718 scope.go:122] "RemoveContainer" containerID="85eb7c970e86320974f0205a2c5dd8c1d03630f6bb715cfc251a3e94c6e9f82f"
Oct 31 02:00:18.808927 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:18.808891    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"]
Oct 31 02:00:19.215248 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.215151    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"]
Oct 31 02:00:19.221551 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.221515    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-98b70c-29890200-98s6s"]
Oct 31 02:00:19.334061 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.334020    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"]
Oct 31 02:00:19.337754 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.337726    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890200-qgksm"]
Oct 31 02:00:19.720494 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.720460    5718 generic.go:356] "Generic (PLEG): container finished" podID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerID="6750a8558576cf8ea82589161bfacd2e6cbca32e5abafec0674bf5fbdedd500d" exitCode=60
Oct 31 02:00:19.721000 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.720533    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerDied","Data":"6750a8558576cf8ea82589161bfacd2e6cbca32e5abafec0674bf5fbdedd500d"}
Oct 31 02:00:19.721000 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.720612    5718 scope.go:122] "RemoveContainer" containerID="0dea1ad247f5938bcb6489ab526a018e762af32a048cdfa6d5cd976b1cca9dc2"
Oct 31 02:00:19.721139 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.721021    5718 scope.go:122] "RemoveContainer" containerID="6750a8558576cf8ea82589161bfacd2e6cbca32e5abafec0674bf5fbdedd500d"
Oct 31 02:00:19.721421 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:19.721388    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj_odh-ai-gateway-infra(94ef7ecc-0532-47b7-8b49-b272d506f569)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569"
Oct 31 02:00:19.722775 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.722746    5718 generic.go:356] "Generic (PLEG): container finished" podID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerID="96847b4c4d349819c66611c34b8155365ecad10ad756f449732e5b11eec31856" exitCode=60
Oct 31 02:00:19.722927 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.722819    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerDied","Data":"96847b4c4d349819c66611c34b8155365ecad10ad756f449732e5b11eec31856"}
Oct 31 02:00:19.723202 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.723186    5718 scope.go:122] "RemoveContainer" containerID="96847b4c4d349819c66611c34b8155365ecad10ad756f449732e5b11eec31856"
Oct 31 02:00:19.723490 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:19.723471    5718 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz_odh-ai-gateway-infra(8d18f623-68e0-41c6-92d8-29e50b8987ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab"
Oct 31 02:00:19.736036 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.736012    5718 scope.go:122] "RemoveContainer" containerID="b99efc09b993d2a896d1161cb6a2c447a36f62a41a8b914d1f69566d2e246d31"
Oct 31 02:00:19.900431 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.900406    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:19.967785 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.967746    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" path="/var/lib/kubelet/pods/6a677ad8-7a45-4180-9075-0ae5caa8b566/volumes"
Oct 31 02:00:19.968268 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.968252    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" path="/var/lib/kubelet/pods/ff8a2102-f8b9-4e38-96bb-48e179759fe8/volumes"
Oct 31 02:00:19.974823 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.974751    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca\") pod \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") "
Oct 31 02:00:19.974959 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.974824    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs\" (UniqueName: \"kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs\") pod \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\" (UID: \"7f44f25a-f1ab-405a-99ce-1e38f335bd83\") "
Oct 31 02:00:19.975088 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.975069    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca" pod "7f44f25a-f1ab-405a-99ce-1e38f335bd83" (UID: "7f44f25a-f1ab-405a-99ce-1e38f335bd83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:19.977212 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.977187    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs" pod "7f44f25a-f1ab-405a-99ce-1e38f335bd83" (UID: "7f44f25a-f1ab-405a-99ce-1e38f335bd83"). InnerVolumeSpecName "kube-api-access-4tbzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:19.997517 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:19.997492    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"]
Oct 31 02:00:20.076416 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.076369    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f44f25a-f1ab-405a-99ce-1e38f335bd83-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.076416 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.076416    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tbzs\" (UniqueName: \"kubernetes.io/projected/7f44f25a-f1ab-405a-99ce-1e38f335bd83-kube-api-access-4tbzs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:20.113365 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.113331    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"]
Oct 31 02:00:20.737332 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.737300    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht" event={"ID":"7f44f25a-f1ab-405a-99ce-1e38f335bd83","Type":"ContainerDied","Data":"0efa096b404a3d23c4ced297ea4f3b7c1d433259381cd4d14a45ce13a89cf93b"}
Oct 31 02:00:20.737682 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.737377    5718 scope.go:122] "RemoveContainer" containerID="a2c84308b44bb2181ca77978698779e2826e47a21caccd0d0e0ca6c06d6849c9"
Oct 31 02:00:20.737682 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.737315    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"
Oct 31 02:00:20.926851 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.926820    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:20.928951 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.928933    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:20.965250 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.965209    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"]
Oct 31 02:00:20.970451 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:20.970423    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890200-st9ht"]
Oct 31 02:00:21.090265 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090233    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca\") pod \"8d18f623-68e0-41c6-92d8-29e50b8987ab\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") "
Oct 31 02:00:21.090265 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090271    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5\" (UniqueName: \"kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5\") pod \"94ef7ecc-0532-47b7-8b49-b272d506f569\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") "
Oct 31 02:00:21.090539 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090307    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87\" (UniqueName: \"kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87\") pod \"8d18f623-68e0-41c6-92d8-29e50b8987ab\" (UID: \"8d18f623-68e0-41c6-92d8-29e50b8987ab\") "
Oct 31 02:00:21.090539 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090335    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca\") pod \"94ef7ecc-0532-47b7-8b49-b272d506f569\" (UID: \"94ef7ecc-0532-47b7-8b49-b272d506f569\") "
Oct 31 02:00:21.090708 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090684    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca" pod "8d18f623-68e0-41c6-92d8-29e50b8987ab" (UID: "8d18f623-68e0-41c6-92d8-29e50b8987ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.090828 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090803    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca" pod "94ef7ecc-0532-47b7-8b49-b272d506f569" (UID: "94ef7ecc-0532-47b7-8b49-b272d506f569"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.090939 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090924    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d18f623-68e0-41c6-92d8-29e50b8987ab-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.091010 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.090941    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ef7ecc-0532-47b7-8b49-b272d506f569-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.092559 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.092533    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5" pod "94ef7ecc-0532-47b7-8b49-b272d506f569" (UID: "94ef7ecc-0532-47b7-8b49-b272d506f569"). InnerVolumeSpecName "kube-api-access-nw9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.092737 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.092721    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87" pod "8d18f623-68e0-41c6-92d8-29e50b8987ab" (UID: "8d18f623-68e0-41c6-92d8-29e50b8987ab"). InnerVolumeSpecName "kube-api-access-9rw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.192263 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.192231    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rw87\" (UniqueName: \"kubernetes.io/projected/8d18f623-68e0-41c6-92d8-29e50b8987ab-kube-api-access-9rw87\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.192263 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.192264    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9t5\" (UniqueName: \"kubernetes.io/projected/94ef7ecc-0532-47b7-8b49-b272d506f569-kube-api-access-nw9t5\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.744891 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.744865    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"
Oct 31 02:00:21.745374 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.744892    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj" event={"ID":"94ef7ecc-0532-47b7-8b49-b272d506f569","Type":"ContainerDied","Data":"451ce77ed9bb7d5c25a90dc32de88f3cd6cb75c206d40151058a383a3e0ae6a6"}
Oct 31 02:00:21.745374 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.744968    5718 scope.go:122] "RemoveContainer" containerID="6750a8558576cf8ea82589161bfacd2e6cbca32e5abafec0674bf5fbdedd500d"
Oct 31 02:00:21.746349 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.746326    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"
Oct 31 02:00:21.746349 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.746336    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz" event={"ID":"8d18f623-68e0-41c6-92d8-29e50b8987ab","Type":"ContainerDied","Data":"b079b1256915936a78b2c286a1ce16a262a941b7651bc8f7325decd2c8c1e9d9"}
Oct 31 02:00:21.755405 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.755385    5718 scope.go:122] "RemoveContainer" containerID="96847b4c4d349819c66611c34b8155365ecad10ad756f449732e5b11eec31856"
Oct 31 02:00:21.909864 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.909822    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"]
Oct 31 02:00:21.913985 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.913955    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9e3b95-29890200-qgmvz"]
Oct 31 02:00:21.974699 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.974644    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" path="/var/lib/kubelet/pods/7f44f25a-f1ab-405a-99ce-1e38f335bd83/volumes"
Oct 31 02:00:21.975130 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:21.975114    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" path="/var/lib/kubelet/pods/8d18f623-68e0-41c6-92d8-29e50b8987ab/volumes"
Oct 31 02:00:22.026050 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:22.026006    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"]
Oct 31 02:00:22.029769 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:22.029740    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-98c748-29890200-d4jvj"]
Oct 31 02:00:23.968197 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:23.968118    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" path="/var/lib/kubelet/pods/94ef7ecc-0532-47b7-8b49-b272d506f569/volumes"
Oct 31 02:00:37.926470 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:00:37.924691    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:00:41.709273 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:41.709235    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"]
Oct 31 02:00:41.709856 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:41.709539    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy" containerID="cri-o://26a25672bb38d68636abd6f7ec36823779b04b09fdfdbb9bcbc72229d607d51c" gracePeriod=30
Oct 31 02:00:46.915038 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:46.914807    5718 generic.go:356] "Generic (PLEG): container finished" podID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerID="26a25672bb38d68636abd6f7ec36823779b04b09fdfdbb9bcbc72229d607d51c" exitCode=0
Oct 31 02:00:46.915038 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:46.914927    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" event={"ID":"3e5a7932-7467-41e5-86df-e2ec9ecd4610","Type":"ContainerDied","Data":"26a25672bb38d68636abd6f7ec36823779b04b09fdfdbb9bcbc72229d607d51c"}
Oct 31 02:00:47.001887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.001857    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 02:00:47.105034 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.104996    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105239 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105042    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105239 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105071    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105239 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105130    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105239 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105167    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105239 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105207    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105503 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105269    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105503 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105373    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105503 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105409    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket\") pod \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\" (UID: \"3e5a7932-7467-41e5-86df-e2ec9ecd4610\") "
Oct 31 02:00:47.105503 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105436    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.105717 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105503    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.105717 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105696    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:47.105942 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105914    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.106014 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.105981    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.106014 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.106002    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.106115 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.106015    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.106317 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.106259    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.107848 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.107819    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.107983 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.107932    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "kube-api-access-jc5cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.108136 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.108109    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:47.108227 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.108146    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy" pod "3e5a7932-7467-41e5-86df-e2ec9ecd4610" (UID: "3e5a7932-7467-41e5-86df-e2ec9ecd4610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.207563 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.207525    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5cs\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-kube-api-access-jc5cs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.207563 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.207555    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.207563 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.207566    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.207563 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.207574    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.207971 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.207584    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.207971 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.207592    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e5a7932-7467-41e5-86df-e2ec9ecd4610-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.922029 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.921996    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"
Oct 31 02:00:47.922494 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.922033    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm" event={"ID":"3e5a7932-7467-41e5-86df-e2ec9ecd4610","Type":"ContainerDied","Data":"e3744d10ff78127d21f8f3edebd8fab87900b4241019c620fb7d1461c5731820"}
Oct 31 02:00:47.922494 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:47.922122    5718 scope.go:122] "RemoveContainer" containerID="26a25672bb38d68636abd6f7ec36823779b04b09fdfdbb9bcbc72229d607d51c"
Oct 31 02:00:48.093684 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:48.093629    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"]
Oct 31 02:00:48.144207 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:48.144172    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-hrtsm"]
Oct 31 02:00:49.967747 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:00:49.967715    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" path="/var/lib/kubelet/pods/3e5a7932-7467-41e5-86df-e2ec9ecd4610/volumes"
Oct 31 02:01:07.924360 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:01:07.924324    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:01:37.923474 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:01:37.923435    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:02:07.924500 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:02:07.924413    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:02:37.924453 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:02:37.924408    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:02:56.498427 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.498385    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"]
Oct 31 02:02:56.498903 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.498636    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerName="discovery" containerID="cri-o://b369f8a15a968a6de76826d9d3a38d985077e00e7f753a39daf946f01290d1cf" gracePeriod=30
Oct 31 02:02:56.684682 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.684339    5718 generic.go:356] "Generic (PLEG): container finished" podID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerID="b369f8a15a968a6de76826d9d3a38d985077e00e7f753a39daf946f01290d1cf" exitCode=0
Oct 31 02:02:56.684682 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.684467    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" event={"ID":"2eb553ac-5f6d-48f8-9496-5b6b22e92314","Type":"ContainerDied","Data":"b369f8a15a968a6de76826d9d3a38d985077e00e7f753a39daf946f01290d1cf"}
Oct 31 02:02:56.776434 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.776408    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 02:02:56.878040 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878006    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878040 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878046    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878291 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878120    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878291 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878158    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878291 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878204    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878291 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878255    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878497 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878304    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig\") pod \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\" (UID: \"2eb553ac-5f6d-48f8-9496-5b6b22e92314\") "
Oct 31 02:02:56.878696 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878614    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:02:56.879049 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.878860    5718 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-ca-configmap\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:56.880684 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.880609    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:02:56.880795 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.880750    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:02:56.881126 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.881106    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:56.881211 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.881143    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "kube-api-access-d642g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:56.881256 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.881229    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:02:56.881297 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.881246    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts" pod "2eb553ac-5f6d-48f8-9496-5b6b22e92314" (UID: "2eb553ac-5f6d-48f8-9496-5b6b22e92314"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:02:56.980605 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.980557    5718 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-csr-dns-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:56.980605 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.980597    5718 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eb553ac-5f6d-48f8-9496-5b6b22e92314-local-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:56.980605 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.980608    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d642g\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-kube-api-access-d642g\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:56.980605 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.980618    5718 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-cacerts\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:56.980925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.980626    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:56.980925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:56.980635    5718 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2eb553ac-5f6d-48f8-9496-5b6b22e92314-istio-kubeconfig\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:57.692824 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:57.692794    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"
Oct 31 02:02:57.692824 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:57.692822    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" event={"ID":"2eb553ac-5f6d-48f8-9496-5b6b22e92314","Type":"ContainerDied","Data":"7f9643ec7b54d4275135071c1307dc36cf712b75253ddd796aa149658e8885eb"}
Oct 31 02:02:57.693415 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:57.692887    5718 scope.go:122] "RemoveContainer" containerID="b369f8a15a968a6de76826d9d3a38d985077e00e7f753a39daf946f01290d1cf"
Oct 31 02:02:57.909379 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:57.909340    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"]
Oct 31 02:02:57.915155 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:57.915122    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m"]
Oct 31 02:02:57.968352 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:02:57.968268    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" path="/var/lib/kubelet/pods/2eb553ac-5f6d-48f8-9496-5b6b22e92314/volumes"
Oct 31 02:03:01.733379 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:01.733347    5718 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lbd4m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 02:03:01.733896 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:01.733410    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbd4m" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.103:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:03:06.228244 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228210    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"]
Oct 31 02:03:06.228749 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228734    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228750    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228770    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228776    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228782    5718 cpu_manager.go:414] "containerMap: removing container" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228787    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228796    5718 cpu_manager.go:414] "containerMap: removing container" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.228810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228802    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228812    5718 cpu_manager.go:414] "containerMap: removing container" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228820    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228834    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228842    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228862    5718 cpu_manager.go:414] "containerMap: removing container" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228866    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228878    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228882    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228894    5718 cpu_manager.go:414] "containerMap: removing container" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228898    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228909    5718 cpu_manager.go:414] "containerMap: removing container" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerName="discovery"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228916    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerName="discovery"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228925    5718 cpu_manager.go:414] "containerMap: removing container" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228930    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228938    5718 cpu_manager.go:414] "containerMap: removing container" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228942    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228949    5718 cpu_manager.go:414] "containerMap: removing container" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228954    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228963    5718 cpu_manager.go:414] "containerMap: removing container" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228967    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228973    5718 cpu_manager.go:414] "containerMap: removing container" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228978    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.228999    5718 cpu_manager.go:414] "containerMap: removing container" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229004    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229085    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229094    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229100    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229106    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d18f623-68e0-41c6-92d8-29e50b8987ab" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229112    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229117    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229120    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229125    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb553ac-5f6d-48f8-9496-5b6b22e92314" containerName="discovery"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229130    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229134    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229138    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5a7932-7467-41e5-86df-e2ec9ecd4610" containerName="istio-proxy"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229148    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229156    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229161    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f44f25a-f1ab-405a-99ce-1e38f335bd83" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229167    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ef7ecc-0532-47b7-8b49-b272d506f569" containerName="cleanup"
Oct 31 02:03:06.229750 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.229173    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:06.234023 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.234008    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.286829 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.286798    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-5632ea\"/\"openshift-service-ca.crt\""
Oct 31 02:03:06.286829 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.286823    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-5632ea\"/\"kube-root-ca.crt\""
Oct 31 02:03:06.291542 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.291511    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.291716 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.291584    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.291716 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.291616    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.291842 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.291798    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p9k\" (UniqueName: \"kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.291943 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.291923    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.291999 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.291960    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.337396 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.337352    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-5632ea\"/\"auth-test-model-5632ea-kserve-self-signed-certs\""
Oct 31 02:03:06.337396 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.337372    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-5632ea\"/\"default-dockercfg-llns4\""
Oct 31 02:03:06.341490 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.341461    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"]
Oct 31 02:03:06.393073 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393027    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393399 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393087    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393399 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393200    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p9k\" (UniqueName: \"kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393399 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393304    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393399 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393342    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393695 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393518    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393695 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393573    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393837 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393739    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.393837 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.393767    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.396080 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.396050    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.396187 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.396054    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.452719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.452684    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p9k\" (UniqueName: \"kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k\") pod \"auth-test-model-5632ea-kserve-586585f977-r9jzq\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") " pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.546644 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.546602    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:06.935521 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:06.935492    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"]
Oct 31 02:03:06.937048 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:03:06.937017    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46e7fb86_a7f6_4fd4_ae78_7d6e879f4ff5.slice/crio-4fd5de5e4d0ec422688ceb38c6a5c075d7a14176a8a5b2acc12232ea413db497 WatchSource:0}: Error finding container 4fd5de5e4d0ec422688ceb38c6a5c075d7a14176a8a5b2acc12232ea413db497: Status 404 returned error can't find the container with id 4fd5de5e4d0ec422688ceb38c6a5c075d7a14176a8a5b2acc12232ea413db497
Oct 31 02:03:07.760020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:07.759978    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq" event={"ID":"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5","Type":"ContainerStarted","Data":"f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556"}
Oct 31 02:03:07.760527 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:07.760030    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq" event={"ID":"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5","Type":"ContainerStarted","Data":"4fd5de5e4d0ec422688ceb38c6a5c075d7a14176a8a5b2acc12232ea413db497"}
Oct 31 02:03:07.879837 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:07.879778    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq" podStartSLOduration=1.879760221 podStartE2EDuration="1.879760221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:03:07.878351408 +0000 UTC m=+3504.041494824" watchObservedRunningTime="2026-10-31 02:03:07.879760221 +0000 UTC m=+3504.042903636"
Oct 31 02:03:07.925455 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:03:07.925417    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:03:08.547537 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:08.547498    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:12.846604 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:12.846529    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"]
Oct 31 02:03:12.847173 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:12.846757    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy" containerID="cri-o://1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907" gracePeriod=30
Oct 31 02:03:16.547617 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:16.547582    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:16.561195 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:16.561167    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:16.823768 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:16.823684    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:03:18.121408 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.121340    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 02:03:18.271781 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.271742    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.271973 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.271810    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.271973 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.271842    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.271973 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.271923    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.271973 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.271944    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.272184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.271982    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.272184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272049    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.272184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272101    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.272184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272106    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:18.272184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272141    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo\") pod \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\" (UID: \"246c83a2-0d9e-4460-ae0c-9601b2cfa293\") "
Oct 31 02:03:18.272184 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272135    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:18.272498 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272169    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:18.272498 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272313    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:18.272498 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272425    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:03:18.272716 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272647    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.272716 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272707    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.273003 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272720    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.273003 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272736    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.273003 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.272749    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.274475 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.274441    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:18.274598 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.274505    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:18.274598 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.274523    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "kube-api-access-hnwkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:18.274598 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.274552    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo" pod "246c83a2-0d9e-4460-ae0c-9601b2cfa293" (UID: "246c83a2-0d9e-4460-ae0c-9601b2cfa293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:03:18.374296 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.374208    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.374296 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.374243    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwkz\" (UniqueName: \"kubernetes.io/projected/246c83a2-0d9e-4460-ae0c-9601b2cfa293-kube-api-access-hnwkz\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.374296 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.374254    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.374296 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.374263    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246c83a2-0d9e-4460-ae0c-9601b2cfa293-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:18.827534 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.827496    5718 generic.go:356] "Generic (PLEG): container finished" podID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerID="1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907" exitCode=0
Oct 31 02:03:18.827762 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.827551    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"
Oct 31 02:03:18.827762 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.827587    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" event={"ID":"246c83a2-0d9e-4460-ae0c-9601b2cfa293","Type":"ContainerDied","Data":"1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907"}
Oct 31 02:03:18.827762 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.827685    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz" event={"ID":"246c83a2-0d9e-4460-ae0c-9601b2cfa293","Type":"ContainerDied","Data":"49d9efbcc999423a48196cef820480e8851efa61e8a1712bba813b3e88299473"}
Oct 31 02:03:18.827762 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.827717    5718 scope.go:122] "RemoveContainer" containerID="1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907"
Oct 31 02:03:18.838792 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.838775    5718 scope.go:122] "RemoveContainer" containerID="1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907"
Oct 31 02:03:18.839071 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:03:18.839053    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907\": container with ID starting with 1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907 not found: ID does not exist" containerID="1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907"
Oct 31 02:03:18.839147 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.839079    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907"} err="failed to get container status \"1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907\": rpc error: code = NotFound desc = could not find container \"1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907\": container with ID starting with 1c5f39a365d86ad85dc74ccb7411c48c7375dd040cb07c7149b2392f324ff907 not found: ID does not exist"
Oct 31 02:03:18.993830 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:18.993789    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"]
Oct 31 02:03:19.044695 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:19.044632    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-6fpwz"]
Oct 31 02:03:19.967730 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:19.967694    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" path="/var/lib/kubelet/pods/246c83a2-0d9e-4460-ae0c-9601b2cfa293/volumes"
Oct 31 02:03:37.925874 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:03:37.925832    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:03:47.256732 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.256691    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"]
Oct 31 02:03:47.257171 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.257158    5718 cpu_manager.go:414] "containerMap: removing container" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy"
Oct 31 02:03:47.257220 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.257171    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy"
Oct 31 02:03:47.257220 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.257201    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:47.257220 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.257206    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a677ad8-7a45-4180-9075-0ae5caa8b566" containerName="cleanup"
Oct 31 02:03:47.257317 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.257289    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="246c83a2-0d9e-4460-ae0c-9601b2cfa293" containerName="istio-proxy"
Oct 31 02:03:47.257317 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.257302    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8a2102-f8b9-4e38-96bb-48e179759fe8" containerName="cleanup"
Oct 31 02:03:47.260517 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.260498    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.308727 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.308641    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.308935 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.308808    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.308935 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.308875    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.308935 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.308914    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.309046 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.309032    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.309086 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.309053    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gm2\" (UniqueName: \"kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.318695 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.318641    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"]
Oct 31 02:03:47.366244 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.366201    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c4be41\"/\"kube-root-ca.crt\""
Oct 31 02:03:47.366244 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.366199    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c4be41\"/\"openshift-service-ca.crt\""
Oct 31 02:03:47.366244 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.366245    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c4be41\"/\"default-dockercfg-8652h\""
Oct 31 02:03:47.366559 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.366253    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c4be41\"/\"explicit-model-c4be41-kserve-self-signed-certs\""
Oct 31 02:03:47.410086 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410022    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.410086 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410076    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.410396 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410101    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.410396 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410143    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.410396 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410214    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.410396 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410239    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gm2\" (UniqueName: \"kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.410969 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.410941    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.411164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.411055    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.411164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.411098    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.415628 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.415597    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.425365 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.425332    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.467818 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.467752    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gm2\" (UniqueName: \"kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2\") pod \"explicit-model-c4be41-kserve-589b95f88f-2mrch\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") " pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.573290 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.573182    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:47.975032 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:47.975001    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"]
Oct 31 02:03:47.976265 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:03:47.976241    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14a6aebf_3c12_4ab3_acc7_2581f46f9c32.slice/crio-7c898c7df70af18efc6495da7c9f984ddbabf7f93391562c2403205f7a6483f1 WatchSource:0}: Error finding container 7c898c7df70af18efc6495da7c9f984ddbabf7f93391562c2403205f7a6483f1: Status 404 returned error can't find the container with id 7c898c7df70af18efc6495da7c9f984ddbabf7f93391562c2403205f7a6483f1
Oct 31 02:03:48.020257 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:48.020222    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" event={"ID":"14a6aebf-3c12-4ab3-acc7-2581f46f9c32","Type":"ContainerStarted","Data":"7c898c7df70af18efc6495da7c9f984ddbabf7f93391562c2403205f7a6483f1"}
Oct 31 02:03:49.027428 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:49.027388    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" event={"ID":"14a6aebf-3c12-4ab3-acc7-2581f46f9c32","Type":"ContainerStarted","Data":"b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022"}
Oct 31 02:03:49.145272 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:49.145221    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" podStartSLOduration=2.145204002 podStartE2EDuration="2.145204002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:03:49.14307248 +0000 UTC m=+3545.306215896" watchObservedRunningTime="2026-10-31 02:03:49.145204002 +0000 UTC m=+3545.308347418"
Oct 31 02:03:49.574139 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:49.574096    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:57.574206 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:57.574149    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:57.586921 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:57.586895    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:03:58.091244 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:03:58.091211    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:04:07.925239 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:04:07.925200    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:04:12.082988 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:12.082952    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"]
Oct 31 02:04:12.083678 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:12.083274    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy" containerID="cri-o://55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d" gracePeriod=30
Oct 31 02:04:16.156253 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.156215    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"]
Oct 31 02:04:16.159960 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.159942    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.216323 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.216289    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"]
Oct 31 02:04:16.267957 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.267922    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-183266-dockercfg-dxmnj\""
Oct 31 02:04:16.267957 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.267938    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-183266\""
Oct 31 02:04:16.280527 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.280491    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5zc\" (UniqueName: \"kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc\") pod \"payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.284506 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.284474    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.304497 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.304465    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"]
Oct 31 02:04:16.308219 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.308199    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.365719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.365675    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"]
Oct 31 02:04:16.390453 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.390416    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckl9\" (UniqueName: \"kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9\") pod \"payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") " pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.390618 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.390591    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5zc\" (UniqueName: \"kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc\") pod \"payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.390707 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.390690    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume\") pod \"payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") " pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.390765 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.390731    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.391440 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.391414    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.449294 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.449217    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5zc\" (UniqueName: \"kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc\") pod \"payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.474218 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.474181    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:16.491926 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.491892    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckl9\" (UniqueName: \"kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9\") pod \"payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") " pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.492060 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.492011    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume\") pod \"payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") " pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.492701 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.492676    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume\") pod \"payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") " pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.555607 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.555574    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckl9\" (UniqueName: \"kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9\") pod \"payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") " pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.623663 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.623617    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:16.852720 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.852693    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"]
Oct 31 02:04:16.854549 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:04:16.854519    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4430ffe_4178_4d65_903d_9f7ff0b74391.slice/crio-2b424633babbefe57281031d1a770053edad6d46dad5322c56c4169bd7e86fa6 WatchSource:0}: Error finding container 2b424633babbefe57281031d1a770053edad6d46dad5322c56c4169bd7e86fa6: Status 404 returned error can't find the container with id 2b424633babbefe57281031d1a770053edad6d46dad5322c56c4169bd7e86fa6
Oct 31 02:04:16.856559 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:16.856538    5718 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:04:17.011412 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.011390    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"]
Oct 31 02:04:17.012765 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:04:17.012741    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14ce155_9a74_4b39_a491_9ea3b2270079.slice/crio-41471e9d900467bfbaf215677da3c8a8b6283fafc50a380fa00543d4d33bf946 WatchSource:0}: Error finding container 41471e9d900467bfbaf215677da3c8a8b6283fafc50a380fa00543d4d33bf946: Status 404 returned error can't find the container with id 41471e9d900467bfbaf215677da3c8a8b6283fafc50a380fa00543d4d33bf946
Oct 31 02:04:17.194868 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.194781    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q" event={"ID":"c14ce155-9a74-4b39-a491-9ea3b2270079","Type":"ContainerStarted","Data":"41471e9d900467bfbaf215677da3c8a8b6283fafc50a380fa00543d4d33bf946"}
Oct 31 02:04:17.196164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.196136    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk" event={"ID":"f4430ffe-4178-4d65-903d-9f7ff0b74391","Type":"ContainerStarted","Data":"2b424633babbefe57281031d1a770053edad6d46dad5322c56c4169bd7e86fa6"}
Oct 31 02:04:17.341092 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.341066    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 02:04:17.402611 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402573    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.402841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402631    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.402841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402681    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.402841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402710    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.402841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402835    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.403078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402907    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.403078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402966    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.403078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.402994    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.403078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403007    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.403078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403031    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.403078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403068    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy\") pod \"5a56e246-18a3-4aae-a647-0329714ec549\" (UID: \"5a56e246-18a3-4aae-a647-0329714ec549\") "
Oct 31 02:04:17.403404 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403206    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.403404 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403300    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.403404 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403314    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:17.403708 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403685    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.403708 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403712    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a56e246-18a3-4aae-a647-0329714ec549-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.403708 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403726    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.403708 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403739    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.403708 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.403752    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.405226 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.405205    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:04:17.405398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.405377    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:17.405704 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.405683    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.409730 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.405877    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs" pod "5a56e246-18a3-4aae-a647-0329714ec549" (UID: "5a56e246-18a3-4aae-a647-0329714ec549"). InnerVolumeSpecName "kube-api-access-6mjgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:17.505133 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.505094    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a56e246-18a3-4aae-a647-0329714ec549-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.505133 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.505128    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.505133 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.505142    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a56e246-18a3-4aae-a647-0329714ec549-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.505388 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:17.505182    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mjgs\" (UniqueName: \"kubernetes.io/projected/5a56e246-18a3-4aae-a647-0329714ec549-kube-api-access-6mjgs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:18.207725 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.207646    5718 generic.go:356] "Generic (PLEG): container finished" podID="5a56e246-18a3-4aae-a647-0329714ec549" containerID="55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d" exitCode=0
Oct 31 02:04:18.208223 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.207749    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" event={"ID":"5a56e246-18a3-4aae-a647-0329714ec549","Type":"ContainerDied","Data":"55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d"}
Oct 31 02:04:18.208223 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.207800    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh" event={"ID":"5a56e246-18a3-4aae-a647-0329714ec549","Type":"ContainerDied","Data":"3d8e23a8cced7d88a3a8be04bfd109ec8e6cab4da654c03814858c11030fa350"}
Oct 31 02:04:18.208223 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.207832    5718 scope.go:122] "RemoveContainer" containerID="55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d"
Oct 31 02:04:18.208223 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.208009    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"
Oct 31 02:04:18.221953 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.221506    5718 scope.go:122] "RemoveContainer" containerID="55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d"
Oct 31 02:04:18.222045 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:04:18.221997    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d\": container with ID starting with 55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d not found: ID does not exist" containerID="55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d"
Oct 31 02:04:18.222105 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.222037    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d"} err="failed to get container status \"55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d\": rpc error: code = NotFound desc = could not find container \"55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d\": container with ID starting with 55de4aecd3575c993e0b8119169a39bce7ee8c8964d141da0cad18758abe5e0d not found: ID does not exist"
Oct 31 02:04:18.429934 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.429886    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"]
Oct 31 02:04:18.432334 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:18.432305    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-tr6nh"]
Oct 31 02:04:19.968406 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:19.968375    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a56e246-18a3-4aae-a647-0329714ec549" path="/var/lib/kubelet/pods/5a56e246-18a3-4aae-a647-0329714ec549/volumes"
Oct 31 02:04:20.223538 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:20.223437    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q" event={"ID":"c14ce155-9a74-4b39-a491-9ea3b2270079","Type":"ContainerStarted","Data":"f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d"}
Oct 31 02:04:20.223538 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:20.223505    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:20.225143 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:20.225116    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk" event={"ID":"f4430ffe-4178-4d65-903d-9f7ff0b74391","Type":"ContainerStarted","Data":"8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743"}
Oct 31 02:04:20.225285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:20.225180    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:20.339418 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:20.339350    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q" podStartSLOduration=1.425003488 podStartE2EDuration="4.339337s" totalImagesPullingTime="2.914333512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:20.338051294 +0000 UTC m=+3576.501194712" watchObservedRunningTime="2026-10-31 02:04:20.339337 +0000 UTC m=+3576.502480411"
Oct 31 02:04:20.456256 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:20.456203    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk" podStartSLOduration=1.474335905 podStartE2EDuration="4.456185977s" totalImagesPullingTime="2.981850072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:20.454080236 +0000 UTC m=+3576.617223656" watchObservedRunningTime="2026-10-31 02:04:20.456185977 +0000 UTC m=+3576.619329437"
Oct 31 02:04:31.234114 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:31.234076    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:04:31.234589 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:31.234265    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:04:37.924362 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:04:37.924315    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:04:48.037477 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:04:48.037436    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"]
Oct 31 02:05:03.058566 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.058504    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" podUID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" containerName="main" containerID="cri-o://b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022" gracePeriod=45
Oct 31 02:05:03.080251 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.080222    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" podUID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": dial tcp 10.134.0.112:8000: connect: connection refused"
Oct 31 02:05:03.333977 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.333950    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:05:03.443435 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443351    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm\") pod \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") "
Oct 31 02:05:03.443435 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443429    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs\" (UniqueName: \"kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs\") pod \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") "
Oct 31 02:05:03.443694 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443517    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home\") pod \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") "
Oct 31 02:05:03.443694 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443664    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache\") pod \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") "
Oct 31 02:05:03.443819 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443699    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir\") pod \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") "
Oct 31 02:05:03.443819 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443736    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2\" (UniqueName: \"kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2\") pod \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\" (UID: \"14a6aebf-3c12-4ab3-acc7-2581f46f9c32\") "
Oct 31 02:05:03.443937 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443824    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home" pod "14a6aebf-3c12-4ab3-acc7-2581f46f9c32" (UID: "14a6aebf-3c12-4ab3-acc7-2581f46f9c32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:03.443937 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443908    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache" pod "14a6aebf-3c12-4ab3-acc7-2581f46f9c32" (UID: "14a6aebf-3c12-4ab3-acc7-2581f46f9c32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:03.444035 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.443935    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir" pod "14a6aebf-3c12-4ab3-acc7-2581f46f9c32" (UID: "14a6aebf-3c12-4ab3-acc7-2581f46f9c32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:03.444255 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.444236    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:03.444312 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.444260    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:03.444312 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.444274    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:03.445995 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.445972    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm" pod "14a6aebf-3c12-4ab3-acc7-2581f46f9c32" (UID: "14a6aebf-3c12-4ab3-acc7-2581f46f9c32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:03.446056 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.446027    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs" pod "14a6aebf-3c12-4ab3-acc7-2581f46f9c32" (UID: "14a6aebf-3c12-4ab3-acc7-2581f46f9c32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:03.446092 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.446079    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2" pod "14a6aebf-3c12-4ab3-acc7-2581f46f9c32" (UID: "14a6aebf-3c12-4ab3-acc7-2581f46f9c32"). InnerVolumeSpecName "kube-api-access-j8gm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:03.478681 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.478614    5718 generic.go:356] "Generic (PLEG): container finished" podID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" containerID="b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022" exitCode=0
Oct 31 02:05:03.479057 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.478727    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"
Oct 31 02:05:03.480227 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.478745    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" event={"ID":"14a6aebf-3c12-4ab3-acc7-2581f46f9c32","Type":"ContainerDied","Data":"b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022"}
Oct 31 02:05:03.480227 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.479793    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch" event={"ID":"14a6aebf-3c12-4ab3-acc7-2581f46f9c32","Type":"ContainerDied","Data":"7c898c7df70af18efc6495da7c9f984ddbabf7f93391562c2403205f7a6483f1"}
Oct 31 02:05:03.480227 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.479825    5718 scope.go:122] "RemoveContainer" containerID="b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022"
Oct 31 02:05:03.492406 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.492387    5718 scope.go:122] "RemoveContainer" containerID="b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022"
Oct 31 02:05:03.492772 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:05:03.492752    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022\": container with ID starting with b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022 not found: ID does not exist" containerID="b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022"
Oct 31 02:05:03.492844 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.492782    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022"} err="failed to get container status \"b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022\": rpc error: code = NotFound desc = could not find container \"b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022\": container with ID starting with b986113b000b70a6cdf1fa1f60e2fbe51811a22c9bd618e34eb3d2feebdee022 not found: ID does not exist"
Oct 31 02:05:03.545768 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.545741    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:03.545768 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.545768    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:03.545961 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.545784    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gm2\" (UniqueName: \"kubernetes.io/projected/14a6aebf-3c12-4ab3-acc7-2581f46f9c32-kube-api-access-j8gm2\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:03.645262 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.645227    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"]
Oct 31 02:05:03.649115 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.649089    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c4be41/explicit-model-c4be41-kserve-589b95f88f-2mrch"]
Oct 31 02:05:03.967248 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:03.967216    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" path="/var/lib/kubelet/pods/14a6aebf-3c12-4ab3-acc7-2581f46f9c32/volumes"
Oct 31 02:05:07.924075 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:05:07.924033    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:05:22.175344 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.175303    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"]
Oct 31 02:05:22.177407 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.176142    5718 cpu_manager.go:414] "containerMap: removing container" podUID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" containerName="main"
Oct 31 02:05:22.177407 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.176163    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" containerName="main"
Oct 31 02:05:22.177407 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.176182    5718 cpu_manager.go:414] "containerMap: removing container" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy"
Oct 31 02:05:22.177407 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.176189    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy"
Oct 31 02:05:22.177407 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.176343    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a6aebf-3c12-4ab3-acc7-2581f46f9c32" containerName="main"
Oct 31 02:05:22.177407 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.176356    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a56e246-18a3-4aae-a647-0329714ec549" containerName="istio-proxy"
Oct 31 02:05:22.180123 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.180106    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.233191 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.233152    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-183266-serving-cert\""
Oct 31 02:05:22.267580 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.267542    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.267791 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.267678    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.267868 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.267828    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dpt\" (UniqueName: \"kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.267928 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.267879    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.286182 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.286145    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"]
Oct 31 02:05:22.368861 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.368827    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dpt\" (UniqueName: \"kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.369032 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.368870    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.369032 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.368949    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.369032 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.368991    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.369157 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:05:22.369122    5718 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-183266-serving-cert: secret "maas-api-e2e-autores-183266-serving-cert" not found
Oct 31 02:05:22.369209 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:05:22.369198    5718 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls podName:f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0 nodeName:}" failed. No retries permitted until 2026-10-31 02:05:22.869176409 +0000 UTC m=+3639.032319802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls") pod "maas-api-e2e-autores-183266-bd4475b97-hq64m" (UID: "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0") : secret "maas-api-e2e-autores-183266-serving-cert" not found
Oct 31 02:05:22.369632 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.369611    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.371607 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.371577    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.426884 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.426806    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dpt\" (UniqueName: \"kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.874340 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.874303    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:22.876918 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:22.876889    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\") pod \"maas-api-e2e-autores-183266-bd4475b97-hq64m\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:23.095933 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:23.095897    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:23.478722 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:23.478687    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"]
Oct 31 02:05:23.486843 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:05:23.486813    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d94c9c_a085_4f7b_8022_a26c2b9ea8a0.slice/crio-06862ec187d0aa8380db21cd6ed0ad611bf2e4815e6ff91fba234bfc2e2aee78 WatchSource:0}: Error finding container 06862ec187d0aa8380db21cd6ed0ad611bf2e4815e6ff91fba234bfc2e2aee78: Status 404 returned error can't find the container with id 06862ec187d0aa8380db21cd6ed0ad611bf2e4815e6ff91fba234bfc2e2aee78
Oct 31 02:05:23.597498 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:23.597462    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m" event={"ID":"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0","Type":"ContainerStarted","Data":"06862ec187d0aa8380db21cd6ed0ad611bf2e4815e6ff91fba234bfc2e2aee78"}
Oct 31 02:05:24.604848 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:24.604756    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m" event={"ID":"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0","Type":"ContainerStarted","Data":"b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba"}
Oct 31 02:05:24.605247 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:24.604965    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:24.723187 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:24.723135    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m" podStartSLOduration=1.8829606110000001 podStartE2EDuration="2.723120544s" totalImagesPullingTime="840.159933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:24.720312995 +0000 UTC m=+3640.883456410" watchObservedRunningTime="2026-10-31 02:05:24.723120544 +0000 UTC m=+3640.886263959"
Oct 31 02:05:30.616844 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:30.616817    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:05:37.923514 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:05:37.923476    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:05:58.496062 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.496028    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"]
Oct 31 02:05:58.500035 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.500011    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.591212 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.591175    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.591212 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.591216    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.591499 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.591350    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.591499 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.591475    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.591591 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.591536    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj57p\" (UniqueName: \"kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.591591 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.591559    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.604319 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.604298    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8114c9\"/\"openshift-service-ca.crt\""
Oct 31 02:05:58.604484 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.604437    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8114c9\"/\"noten-model-8114c9-kserve-self-signed-certs\""
Oct 31 02:05:58.604484 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.604444    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8114c9\"/\"kube-root-ca.crt\""
Oct 31 02:05:58.604593 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.604543    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8114c9\"/\"default-dockercfg-n5hk7\""
Oct 31 02:05:58.606863 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.606840    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"]
Oct 31 02:05:58.692216 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692182    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj57p\" (UniqueName: \"kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692400 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692224    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692400 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692313    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692400 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692338    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692569 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692420    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692569 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692509    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692773 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692752    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692853 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692827    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.692912 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.692873    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.695026 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.695002    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.695224 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.695209    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.754334 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.754260    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj57p\" (UniqueName: \"kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p\") pod \"noten-model-8114c9-kserve-699d56655c-l5fj5\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") " pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:58.812313 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:58.812275    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:59.193518 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:59.191454    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"]
Oct 31 02:05:59.195441 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:05:59.195397    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd660f4e0_a3e1_4095_a42e_b2924b01071c.slice/crio-62fd0b6aafa459a3dc13c2abb427215e5c296f50a0b5a63e3d97937fdeece93f WatchSource:0}: Error finding container 62fd0b6aafa459a3dc13c2abb427215e5c296f50a0b5a63e3d97937fdeece93f: Status 404 returned error can't find the container with id 62fd0b6aafa459a3dc13c2abb427215e5c296f50a0b5a63e3d97937fdeece93f
Oct 31 02:05:59.811631 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:59.811592    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5" event={"ID":"d660f4e0-a3e1-4095-a42e-b2924b01071c","Type":"ContainerStarted","Data":"df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c"}
Oct 31 02:05:59.811631 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:59.811636    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5" event={"ID":"d660f4e0-a3e1-4095-a42e-b2924b01071c","Type":"ContainerStarted","Data":"62fd0b6aafa459a3dc13c2abb427215e5c296f50a0b5a63e3d97937fdeece93f"}
Oct 31 02:05:59.812907 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:59.812885    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:05:59.927414 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:05:59.927356    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5" podStartSLOduration=1.927343475 podStartE2EDuration="1.927343475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:59.926281823 +0000 UTC m=+3676.089425243" watchObservedRunningTime="2026-10-31 02:05:59.927343475 +0000 UTC m=+3676.090486890"
Oct 31 02:06:07.924235 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:06:07.924185    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:06:08.813405 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:08.813363    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:06:08.826016 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:08.825989    5718 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:06:08.875465 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:08.875437    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:06:31.046868 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:31.046818    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"]
Oct 31 02:06:37.926502 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:06:37.926460    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:06:46.068792 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.068715    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5" podUID="d660f4e0-a3e1-4095-a42e-b2924b01071c" containerName="main" containerID="cri-o://df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c" gracePeriod=45
Oct 31 02:06:46.341213 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.341144    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:06:46.364743 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.364707    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm\") pod \"d660f4e0-a3e1-4095-a42e-b2924b01071c\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") "
Oct 31 02:06:46.364926 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.364876    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home\") pod \"d660f4e0-a3e1-4095-a42e-b2924b01071c\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") "
Oct 31 02:06:46.364926 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.364919    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs\") pod \"d660f4e0-a3e1-4095-a42e-b2924b01071c\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") "
Oct 31 02:06:46.365035 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.364965    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p\" (UniqueName: \"kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p\") pod \"d660f4e0-a3e1-4095-a42e-b2924b01071c\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") "
Oct 31 02:06:46.365035 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365021    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache\") pod \"d660f4e0-a3e1-4095-a42e-b2924b01071c\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") "
Oct 31 02:06:46.365151 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365071    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir\") pod \"d660f4e0-a3e1-4095-a42e-b2924b01071c\" (UID: \"d660f4e0-a3e1-4095-a42e-b2924b01071c\") "
Oct 31 02:06:46.365209 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365180    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home" pod "d660f4e0-a3e1-4095-a42e-b2924b01071c" (UID: "d660f4e0-a3e1-4095-a42e-b2924b01071c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:46.365354 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365322    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache" pod "d660f4e0-a3e1-4095-a42e-b2924b01071c" (UID: "d660f4e0-a3e1-4095-a42e-b2924b01071c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:46.365454 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365353    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir" pod "d660f4e0-a3e1-4095-a42e-b2924b01071c" (UID: "d660f4e0-a3e1-4095-a42e-b2924b01071c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:46.365626 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365605    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:46.365731 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365632    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:46.365731 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.365645    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:46.367523 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.367494    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs" pod "d660f4e0-a3e1-4095-a42e-b2924b01071c" (UID: "d660f4e0-a3e1-4095-a42e-b2924b01071c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:46.367689 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.367644    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm" pod "d660f4e0-a3e1-4095-a42e-b2924b01071c" (UID: "d660f4e0-a3e1-4095-a42e-b2924b01071c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:46.367805 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.367694    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p" pod "d660f4e0-a3e1-4095-a42e-b2924b01071c" (UID: "d660f4e0-a3e1-4095-a42e-b2924b01071c"). InnerVolumeSpecName "kube-api-access-xj57p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:46.467285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.467253    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj57p\" (UniqueName: \"kubernetes.io/projected/d660f4e0-a3e1-4095-a42e-b2924b01071c-kube-api-access-xj57p\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:46.467285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.467281    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d660f4e0-a3e1-4095-a42e-b2924b01071c-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:46.467285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:46.467292    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d660f4e0-a3e1-4095-a42e-b2924b01071c-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:47.104874 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.104840    5718 generic.go:356] "Generic (PLEG): container finished" podID="d660f4e0-a3e1-4095-a42e-b2924b01071c" containerID="df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c" exitCode=0
Oct 31 02:06:47.105326 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.104890    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"
Oct 31 02:06:47.105326 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.104927    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5" event={"ID":"d660f4e0-a3e1-4095-a42e-b2924b01071c","Type":"ContainerDied","Data":"df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c"}
Oct 31 02:06:47.105326 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.104977    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5" event={"ID":"d660f4e0-a3e1-4095-a42e-b2924b01071c","Type":"ContainerDied","Data":"62fd0b6aafa459a3dc13c2abb427215e5c296f50a0b5a63e3d97937fdeece93f"}
Oct 31 02:06:47.105326 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.104999    5718 scope.go:122] "RemoveContainer" containerID="df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c"
Oct 31 02:06:47.115134 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.115112    5718 scope.go:122] "RemoveContainer" containerID="df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c"
Oct 31 02:06:47.115426 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:06:47.115408    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c\": container with ID starting with df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c not found: ID does not exist" containerID="df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c"
Oct 31 02:06:47.115471 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.115438    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c"} err="failed to get container status \"df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c\": rpc error: code = NotFound desc = could not find container \"df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c\": container with ID starting with df78e137655041baae9f17d18ab9af19213c7d5cb7ee9f3720a6a286adef972c not found: ID does not exist"
Oct 31 02:06:47.270820 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.270780    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"]
Oct 31 02:06:47.321639 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.321595    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8114c9/noten-model-8114c9-kserve-699d56655c-l5fj5"]
Oct 31 02:06:47.967200 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:06:47.967165    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d660f4e0-a3e1-4095-a42e-b2924b01071c" path="/var/lib/kubelet/pods/d660f4e0-a3e1-4095-a42e-b2924b01071c/volumes"
Oct 31 02:07:07.924350 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:07:07.924310    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:07:29.055474 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:29.055440    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"]
Oct 31 02:07:37.935199 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:07:37.935152    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:07:44.077685 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.077596    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc" podUID="7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" containerName="main" containerID="cri-o://0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617" gracePeriod=45
Oct 31 02:07:44.360085 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.360057    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 02:07:44.449424 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.449390    5718 generic.go:356] "Generic (PLEG): container finished" podID="7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" containerID="0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617" exitCode=0
Oct 31 02:07:44.449641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.449438    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc" event={"ID":"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9","Type":"ContainerDied","Data":"0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617"}
Oct 31 02:07:44.449641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.449454    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"
Oct 31 02:07:44.449641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.449476    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc" event={"ID":"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9","Type":"ContainerDied","Data":"819f67a323af3a86b78b53a5c2d2a63fb0876b192dd7154d2f9a1dcb482da7d1"}
Oct 31 02:07:44.449641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.449493    5718 scope.go:122] "RemoveContainer" containerID="0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617"
Oct 31 02:07:44.452974 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.452949    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home\") pod \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") "
Oct 31 02:07:44.453092 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453005    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache\") pod \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") "
Oct 31 02:07:44.453092 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453055    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4\" (UniqueName: \"kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4\") pod \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") "
Oct 31 02:07:44.453222 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453209    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir\") pod \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") "
Oct 31 02:07:44.453279 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453250    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home" pod "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" (UID: "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.453337 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453276    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs\") pod \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") "
Oct 31 02:07:44.453337 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453284    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache" pod "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" (UID: "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.453434 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453369    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm\") pod \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\" (UID: \"7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9\") "
Oct 31 02:07:44.453485 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453434    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir" pod "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" (UID: "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.453840 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453817    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.453938 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453844    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.453938 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.453859    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.455721 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.455693    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs" pod "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" (UID: "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:44.455851 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.455831    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4" pod "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" (UID: "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9"). InnerVolumeSpecName "kube-api-access-gvxl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:44.457196 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.457171    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm" pod "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" (UID: "7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.460338 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.460317    5718 scope.go:122] "RemoveContainer" containerID="0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617"
Oct 31 02:07:44.460712 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:07:44.460683    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617\": container with ID starting with 0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617 not found: ID does not exist" containerID="0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617"
Oct 31 02:07:44.460795 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.460722    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617"} err="failed to get container status \"0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617\": rpc error: code = NotFound desc = could not find container \"0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617\": container with ID starting with 0ee691e73d9bf5158cca79738be3a8103bc887739ff0a757cdd4140e89f32617 not found: ID does not exist"
Oct 31 02:07:44.555760 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.555720    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.555760 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.555760    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxl4\" (UniqueName: \"kubernetes.io/projected/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-kube-api-access-gvxl4\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.556035 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.555777    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.875071 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.875034    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"]
Oct 31 02:07:44.928944 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:44.925897    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-2-w4-8f6b5b-kserve-68f5bd8b4c-t9zxc"]
Oct 31 02:07:45.967223 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:07:45.967181    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" path="/var/lib/kubelet/pods/7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9/volumes"
Oct 31 02:08:07.923644 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:07.923548    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:08:28.997566 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:28.997526    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"]
Oct 31 02:08:28.998135 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:28.997809    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr" podUID="145e629f-8852-4a5e-9286-5ad9e17eb96e" containerName="maas-api" containerID="cri-o://0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6" gracePeriod=30
Oct 31 02:08:29.292575 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.292537    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 02:08:29.359593 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.359558    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls\") pod \"145e629f-8852-4a5e-9286-5ad9e17eb96e\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") "
Oct 31 02:08:29.359593 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.359592    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca\") pod \"145e629f-8852-4a5e-9286-5ad9e17eb96e\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") "
Oct 31 02:08:29.359864 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.359680    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4\" (UniqueName: \"kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4\") pod \"145e629f-8852-4a5e-9286-5ad9e17eb96e\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") "
Oct 31 02:08:29.359864 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.359703    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs\") pod \"145e629f-8852-4a5e-9286-5ad9e17eb96e\" (UID: \"145e629f-8852-4a5e-9286-5ad9e17eb96e\") "
Oct 31 02:08:29.360141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.360077    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca" pod "145e629f-8852-4a5e-9286-5ad9e17eb96e" (UID: "145e629f-8852-4a5e-9286-5ad9e17eb96e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:29.360299 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.360266    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145e629f-8852-4a5e-9286-5ad9e17eb96e-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.362091 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.362061    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4" pod "145e629f-8852-4a5e-9286-5ad9e17eb96e" (UID: "145e629f-8852-4a5e-9286-5ad9e17eb96e"). InnerVolumeSpecName "kube-api-access-895h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:29.362454 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.362431    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls" pod "145e629f-8852-4a5e-9286-5ad9e17eb96e" (UID: "145e629f-8852-4a5e-9286-5ad9e17eb96e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:29.362549 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.362458    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs" pod "145e629f-8852-4a5e-9286-5ad9e17eb96e" (UID: "145e629f-8852-4a5e-9286-5ad9e17eb96e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:29.461758 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.461721    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.461758 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.461755    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895h4\" (UniqueName: \"kubernetes.io/projected/145e629f-8852-4a5e-9286-5ad9e17eb96e-kube-api-access-895h4\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.461758 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.461766    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145e629f-8852-4a5e-9286-5ad9e17eb96e-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:29.716643 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.716564    5718 generic.go:356] "Generic (PLEG): container finished" podID="145e629f-8852-4a5e-9286-5ad9e17eb96e" containerID="0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6" exitCode=0
Oct 31 02:08:29.716643 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.716613    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"
Oct 31 02:08:29.716919 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.716681    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr" event={"ID":"145e629f-8852-4a5e-9286-5ad9e17eb96e","Type":"ContainerDied","Data":"0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6"}
Oct 31 02:08:29.716919 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.716740    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr" event={"ID":"145e629f-8852-4a5e-9286-5ad9e17eb96e","Type":"ContainerDied","Data":"2490472ade3f4d79207e9c46f68f2406483e3af311c3e0d6378fe95666505f35"}
Oct 31 02:08:29.716919 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.716769    5718 scope.go:122] "RemoveContainer" containerID="0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6"
Oct 31 02:08:29.727710 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.727693    5718 scope.go:122] "RemoveContainer" containerID="0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6"
Oct 31 02:08:29.727985 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:29.727965    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6\": container with ID starting with 0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6 not found: ID does not exist" containerID="0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6"
Oct 31 02:08:29.728052 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.727993    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6"} err="failed to get container status \"0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6\": rpc error: code = NotFound desc = could not find container \"0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6\": container with ID starting with 0f040f78f487f5d76887b0062e5457037b7884dfa6b49ec7e5f35b882b8aa0a6 not found: ID does not exist"
Oct 31 02:08:29.886078 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.886040    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"]
Oct 31 02:08:29.939758 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.939708    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-98c748-85f9d96697-f4wdr"]
Oct 31 02:08:29.969817 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:29.969736    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145e629f-8852-4a5e-9286-5ad9e17eb96e" path="/var/lib/kubelet/pods/145e629f-8852-4a5e-9286-5ad9e17eb96e/volumes"
Oct 31 02:08:37.923950 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:37.923912    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:08:40.503325 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.503289    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"]
Oct 31 02:08:40.503794 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.503525    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" podUID="b40c8b6a-e6f9-4248-9c10-ec57d60464b0" containerName="maas-api" containerID="cri-o://909b0f9c3e9cf88382218f64c213936fc400b5c010cf15515e2e852b48286d05" gracePeriod=30
Oct 31 02:08:40.785949 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.785918    5718 generic.go:356] "Generic (PLEG): container finished" podID="b40c8b6a-e6f9-4248-9c10-ec57d60464b0" containerID="909b0f9c3e9cf88382218f64c213936fc400b5c010cf15515e2e852b48286d05" exitCode=0
Oct 31 02:08:40.786100 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.785996    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" event={"ID":"b40c8b6a-e6f9-4248-9c10-ec57d60464b0","Type":"ContainerDied","Data":"909b0f9c3e9cf88382218f64c213936fc400b5c010cf15515e2e852b48286d05"}
Oct 31 02:08:40.786100 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.786066    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx" event={"ID":"b40c8b6a-e6f9-4248-9c10-ec57d60464b0","Type":"ContainerDied","Data":"6fde1485eb0c5cf7686bd3a6937fa51332f1867d3e0c368a4e73418b12968d12"}
Oct 31 02:08:40.786100 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.786089    5718 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fde1485eb0c5cf7686bd3a6937fa51332f1867d3e0c368a4e73418b12968d12"
Oct 31 02:08:40.797518 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.797494    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 02:08:40.992804 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.992766    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487\" (UniqueName: \"kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487\") pod \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") "
Oct 31 02:08:40.992992 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.992856    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca\") pod \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") "
Oct 31 02:08:40.992992 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.992946    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\") pod \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") "
Oct 31 02:08:40.992992 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.992979    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs\") pod \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\" (UID: \"b40c8b6a-e6f9-4248-9c10-ec57d60464b0\") "
Oct 31 02:08:40.993573 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.993539    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca" pod "b40c8b6a-e6f9-4248-9c10-ec57d60464b0" (UID: "b40c8b6a-e6f9-4248-9c10-ec57d60464b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:40.995333 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.995304    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs" pod "b40c8b6a-e6f9-4248-9c10-ec57d60464b0" (UID: "b40c8b6a-e6f9-4248-9c10-ec57d60464b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:40.995971 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.995940    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls" pod "b40c8b6a-e6f9-4248-9c10-ec57d60464b0" (UID: "b40c8b6a-e6f9-4248-9c10-ec57d60464b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:40.997016 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:40.996991    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487" pod "b40c8b6a-e6f9-4248-9c10-ec57d60464b0" (UID: "b40c8b6a-e6f9-4248-9c10-ec57d60464b0"). InnerVolumeSpecName "kube-api-access-tq487". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:41.095202 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:41.095125    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq487\" (UniqueName: \"kubernetes.io/projected/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-kube-api-access-tq487\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:41.095202 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:41.095154    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:41.095202 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:41.095164    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:41.095202 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:41.095172    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40c8b6a-e6f9-4248-9c10-ec57d60464b0-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:41.792175 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:41.792140    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"
Oct 31 02:08:41.968558 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:41.968525    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"]
Oct 31 02:08:42.010164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:42.010132    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8f6b5b-6c6bfbc679-rdbwx"]
Oct 31 02:08:43.969107 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:43.969073    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40c8b6a-e6f9-4248-9c10-ec57d60464b0" path="/var/lib/kubelet/pods/b40c8b6a-e6f9-4248-9c10-ec57d60464b0/volumes"
Oct 31 02:08:44.061408 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:44.061376    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"]
Oct 31 02:08:44.115847 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:44.115812    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"]
Oct 31 02:08:44.118088 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:44.118061    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"]
Oct 31 02:08:44.119793 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:44.119769    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"]
Oct 31 02:08:45.004622 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:45.004589    5718 scope.go:122] "RemoveContainer" containerID="909b0f9c3e9cf88382218f64c213936fc400b5c010cf15515e2e852b48286d05"
Oct 31 02:08:49.301194 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:49.301148    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"]
Oct 31 02:08:49.301715 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:49.301434    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy" containerID="cri-o://2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc" gracePeriod=30
Oct 31 02:08:51.719276 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:51.719238    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"]
Oct 31 02:08:51.719785 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:51.719451    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" podUID="b91d1b08-9e21-4a52-a31d-9c7989db39b6" containerName="maas-api" containerID="cri-o://b70ab85d0308d84addc017cd00d1f9da80e054ab8148f766058d0e9fa147e257" gracePeriod=30
Oct 31 02:08:51.854945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:51.854914    5718 generic.go:356] "Generic (PLEG): container finished" podID="b91d1b08-9e21-4a52-a31d-9c7989db39b6" containerID="b70ab85d0308d84addc017cd00d1f9da80e054ab8148f766058d0e9fa147e257" exitCode=0
Oct 31 02:08:51.855099 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:51.854984    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" event={"ID":"b91d1b08-9e21-4a52-a31d-9c7989db39b6","Type":"ContainerDied","Data":"b70ab85d0308d84addc017cd00d1f9da80e054ab8148f766058d0e9fa147e257"}
Oct 31 02:08:51.989365 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:51.989336    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 02:08:52.117196 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.117147    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj\" (UniqueName: \"kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj\") pod \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") "
Oct 31 02:08:52.117390 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.117207    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs\") pod \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") "
Oct 31 02:08:52.117390 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.117250    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca\") pod \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") "
Oct 31 02:08:52.117515 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.117406    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\") pod \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\" (UID: \"b91d1b08-9e21-4a52-a31d-9c7989db39b6\") "
Oct 31 02:08:52.117985 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.117950    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca" pod "b91d1b08-9e21-4a52-a31d-9c7989db39b6" (UID: "b91d1b08-9e21-4a52-a31d-9c7989db39b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:52.120744 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.120714    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj" pod "b91d1b08-9e21-4a52-a31d-9c7989db39b6" (UID: "b91d1b08-9e21-4a52-a31d-9c7989db39b6"). InnerVolumeSpecName "kube-api-access-hxmsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:52.121280 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.121258    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs" pod "b91d1b08-9e21-4a52-a31d-9c7989db39b6" (UID: "b91d1b08-9e21-4a52-a31d-9c7989db39b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:52.124081 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.124056    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls" pod "b91d1b08-9e21-4a52-a31d-9c7989db39b6" (UID: "b91d1b08-9e21-4a52-a31d-9c7989db39b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:52.219573 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.219483    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxmsj\" (UniqueName: \"kubernetes.io/projected/b91d1b08-9e21-4a52-a31d-9c7989db39b6-kube-api-access-hxmsj\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:52.219573 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.219575    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:52.219866 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.219593    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d1b08-9e21-4a52-a31d-9c7989db39b6-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:52.219866 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.219607    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d1b08-9e21-4a52-a31d-9c7989db39b6-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:52.861795 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.861759    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6" event={"ID":"b91d1b08-9e21-4a52-a31d-9c7989db39b6","Type":"ContainerDied","Data":"7dd8d09844f1f4926ebc82ba07018508f86077ce0a7abeb2a86bdbee94803311"}
Oct 31 02:08:52.861795 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.861797    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"
Oct 31 02:08:52.862391 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:52.861831    5718 scope.go:122] "RemoveContainer" containerID="b70ab85d0308d84addc017cd00d1f9da80e054ab8148f766058d0e9fa147e257"
Oct 31 02:08:53.026887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:53.026850    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"]
Oct 31 02:08:53.078689 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:53.078630    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c4d438-7497664fc7-c5xm6"]
Oct 31 02:08:53.967902 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:53.967868    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91d1b08-9e21-4a52-a31d-9c7989db39b6" path="/var/lib/kubelet/pods/b91d1b08-9e21-4a52-a31d-9c7989db39b6/volumes"
Oct 31 02:08:54.569285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.569261    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 02:08:54.646604 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646514    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.646604 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646570    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.646871 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646634    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.646871 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646796    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.646992 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646879    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.646992 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646910    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.646992 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.646922    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.647157 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647044    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.647157 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647136    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.647279 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647176    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket\") pod \"6c783693-7a55-42cf-a4c6-983a680377e9\" (UID: \"6c783693-7a55-42cf-a4c6-983a680377e9\") "
Oct 31 02:08:54.647279 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647188    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.649675 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647518    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.649675 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647605    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.649675 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.647623    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.649675 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.648165    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.649675 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.648414    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:54.649675 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.649453    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:08:54.650334 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.650288    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.650334 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.650310    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "kube-api-access-2kt2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:54.651087 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.651052    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token" pod "6c783693-7a55-42cf-a4c6-983a680377e9" (UID: "6c783693-7a55-42cf-a4c6-983a680377e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:54.749457 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749418    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kt2n\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-kube-api-access-2kt2n\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.749457 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749461    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.749705 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749478    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c783693-7a55-42cf-a4c6-983a680377e9-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.749705 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749492    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.749705 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749505    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c783693-7a55-42cf-a4c6-983a680377e9-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.749705 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749517    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c783693-7a55-42cf-a4c6-983a680377e9-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.749705 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.749527    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c783693-7a55-42cf-a4c6-983a680377e9-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.877013 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.876980    5718 generic.go:356] "Generic (PLEG): container finished" podID="6c783693-7a55-42cf-a4c6-983a680377e9" containerID="2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc" exitCode=0
Oct 31 02:08:54.877210 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.877050    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"
Oct 31 02:08:54.877210 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.877065    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" event={"ID":"6c783693-7a55-42cf-a4c6-983a680377e9","Type":"ContainerDied","Data":"2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc"}
Oct 31 02:08:54.877210 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.877133    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz" event={"ID":"6c783693-7a55-42cf-a4c6-983a680377e9","Type":"ContainerDied","Data":"72ae01fd3289abfcab62c6013fec2963ebd4718873011010c6769f365e5ac1c4"}
Oct 31 02:08:54.877210 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.877162    5718 scope.go:122] "RemoveContainer" containerID="2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc"
Oct 31 02:08:54.888185 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.888165    5718 scope.go:122] "RemoveContainer" containerID="2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc"
Oct 31 02:08:54.888443 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:54.888423    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc\": container with ID starting with 2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc not found: ID does not exist" containerID="2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc"
Oct 31 02:08:54.888522 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:54.888454    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc"} err="failed to get container status \"2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc\": rpc error: code = NotFound desc = could not find container \"2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc\": container with ID starting with 2a0b097271602ce40f8ee27c3f2261d623b316699816f0f333c68063b7470adc not found: ID does not exist"
Oct 31 02:08:55.047471 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:55.047433    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"]
Oct 31 02:08:55.050810 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:55.050782    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-6xnqz"]
Oct 31 02:08:55.967167 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:55.967131    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" path="/var/lib/kubelet/pods/6c783693-7a55-42cf-a4c6-983a680377e9/volumes"
Oct 31 02:08:59.083986 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.083945    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerName="main" containerID="cri-o://0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec" gracePeriod=45
Oct 31 02:08:59.137206 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.137163    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main" containerID="cri-o://337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905" gracePeriod=45
Oct 31 02:08:59.138258 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.138229    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main" containerID="cri-o://a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e" gracePeriod=45
Oct 31 02:08:59.139406 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.139361    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main" containerID="cri-o://138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c" gracePeriod=45
Oct 31 02:08:59.428503 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.427379    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 02:08:59.465604 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.465491    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 02:08:59.475924 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.475897    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 02:08:59.479060 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.479037    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 02:08:59.504327 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504294    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs\") pod \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") "
Oct 31 02:08:59.504504 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504394    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp\" (UniqueName: \"kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp\") pod \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") "
Oct 31 02:08:59.504504 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504450    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home\") pod \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") "
Oct 31 02:08:59.504591 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504544    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm\") pod \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") "
Oct 31 02:08:59.504591 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504583    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache\") pod \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") "
Oct 31 02:08:59.504726 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504621    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir\") pod \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\" (UID: \"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff\") "
Oct 31 02:08:59.504817 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.504786    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home" pod "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" (UID: "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.505035 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.505007    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache" pod "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" (UID: "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.505141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.505072    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.505368 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.505333    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir" pod "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" (UID: "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.506873 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.506847    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp" pod "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" (UID: "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff"). InnerVolumeSpecName "kube-api-access-zkxqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:59.506873 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.506855    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm" pod "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" (UID: "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.507020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.506956    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs" pod "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" (UID: "5fa64d50-608d-4a3d-b5f8-c630ef9c23ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:59.606285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606189    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm\") pod \"3277d75c-b24f-4a79-987e-a2128749a7c7\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") "
Oct 31 02:08:59.606285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606235    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir\") pod \"54f8ef22-0f82-4bb7-aea2-23269ded877d\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") "
Oct 31 02:08:59.606285 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606259    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir\") pod \"b5455c3c-e54d-4088-a012-688845de714a\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606287    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home\") pod \"3277d75c-b24f-4a79-987e-a2128749a7c7\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606309    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm\") pod \"54f8ef22-0f82-4bb7-aea2-23269ded877d\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606380    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache\") pod \"b5455c3c-e54d-4088-a012-688845de714a\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606408    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr\" (UniqueName: \"kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr\") pod \"54f8ef22-0f82-4bb7-aea2-23269ded877d\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606436    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache\") pod \"3277d75c-b24f-4a79-987e-a2128749a7c7\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606474    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home\") pod \"b5455c3c-e54d-4088-a012-688845de714a\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606490    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir" pod "54f8ef22-0f82-4bb7-aea2-23269ded877d" (UID: "54f8ef22-0f82-4bb7-aea2-23269ded877d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606520    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home\") pod \"54f8ef22-0f82-4bb7-aea2-23269ded877d\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") "
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606525    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir" pod "b5455c3c-e54d-4088-a012-688845de714a" (UID: "b5455c3c-e54d-4088-a012-688845de714a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.606582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606552    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home" pod "3277d75c-b24f-4a79-987e-a2128749a7c7" (UID: "3277d75c-b24f-4a79-987e-a2128749a7c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.607141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606745    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache" pod "3277d75c-b24f-4a79-987e-a2128749a7c7" (UID: "3277d75c-b24f-4a79-987e-a2128749a7c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.607141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606838    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home" pod "54f8ef22-0f82-4bb7-aea2-23269ded877d" (UID: "54f8ef22-0f82-4bb7-aea2-23269ded877d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.607141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606852    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache\") pod \"54f8ef22-0f82-4bb7-aea2-23269ded877d\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") "
Oct 31 02:08:59.607141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.606903    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs\") pod \"54f8ef22-0f82-4bb7-aea2-23269ded877d\" (UID: \"54f8ef22-0f82-4bb7-aea2-23269ded877d\") "
Oct 31 02:08:59.607141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607012    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs\") pod \"3277d75c-b24f-4a79-987e-a2128749a7c7\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") "
Oct 31 02:08:59.607141 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607058    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache" pod "54f8ef22-0f82-4bb7-aea2-23269ded877d" (UID: "54f8ef22-0f82-4bb7-aea2-23269ded877d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.607450 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607140    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd\" (UniqueName: \"kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd\") pod \"3277d75c-b24f-4a79-987e-a2128749a7c7\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") "
Oct 31 02:08:59.607450 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607171    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs\") pod \"b5455c3c-e54d-4088-a012-688845de714a\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") "
Oct 31 02:08:59.607450 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607218    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4\" (UniqueName: \"kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4\") pod \"b5455c3c-e54d-4088-a012-688845de714a\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") "
Oct 31 02:08:59.607450 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607249    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm\") pod \"b5455c3c-e54d-4088-a012-688845de714a\" (UID: \"b5455c3c-e54d-4088-a012-688845de714a\") "
Oct 31 02:08:59.607450 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607287    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir\") pod \"3277d75c-b24f-4a79-987e-a2128749a7c7\" (UID: \"3277d75c-b24f-4a79-987e-a2128749a7c7\") "
Oct 31 02:08:59.607838 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607812    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.607922 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607844    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.607922 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607859    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.607922 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607876    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxqp\" (UniqueName: \"kubernetes.io/projected/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-kube-api-access-zkxqp\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.607922 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607890    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.607922 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607903    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.607922 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607914    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.608234 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607926    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.608234 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607939    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.608234 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607951    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.608234 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.607964    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.608437 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.608346    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir" pod "3277d75c-b24f-4a79-987e-a2128749a7c7" (UID: "3277d75c-b24f-4a79-987e-a2128749a7c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.608496 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.608476    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home" pod "b5455c3c-e54d-4088-a012-688845de714a" (UID: "b5455c3c-e54d-4088-a012-688845de714a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.609324 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.609286    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache" pod "b5455c3c-e54d-4088-a012-688845de714a" (UID: "b5455c3c-e54d-4088-a012-688845de714a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.609504 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.609469    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm" pod "3277d75c-b24f-4a79-987e-a2128749a7c7" (UID: "3277d75c-b24f-4a79-987e-a2128749a7c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.609983 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.609956    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm" pod "54f8ef22-0f82-4bb7-aea2-23269ded877d" (UID: "54f8ef22-0f82-4bb7-aea2-23269ded877d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.609983 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.609966    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs" pod "3277d75c-b24f-4a79-987e-a2128749a7c7" (UID: "3277d75c-b24f-4a79-987e-a2128749a7c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:59.610179 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.610154    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr" pod "54f8ef22-0f82-4bb7-aea2-23269ded877d" (UID: "54f8ef22-0f82-4bb7-aea2-23269ded877d"). InnerVolumeSpecName "kube-api-access-fr6wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:59.610730 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.610705    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs" pod "54f8ef22-0f82-4bb7-aea2-23269ded877d" (UID: "54f8ef22-0f82-4bb7-aea2-23269ded877d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:59.611253 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.611229    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd" pod "3277d75c-b24f-4a79-987e-a2128749a7c7" (UID: "3277d75c-b24f-4a79-987e-a2128749a7c7"). InnerVolumeSpecName "kube-api-access-fh9pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:59.611253 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.611238    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs" pod "b5455c3c-e54d-4088-a012-688845de714a" (UID: "b5455c3c-e54d-4088-a012-688845de714a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:59.611494 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.611471    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm" pod "b5455c3c-e54d-4088-a012-688845de714a" (UID: "b5455c3c-e54d-4088-a012-688845de714a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:59.612087 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.612062    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4" pod "b5455c3c-e54d-4088-a012-688845de714a" (UID: "b5455c3c-e54d-4088-a012-688845de714a"). InnerVolumeSpecName "kube-api-access-dwfq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:59.711290 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711259    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711504 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711484    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6wr\" (UniqueName: \"kubernetes.io/projected/54f8ef22-0f82-4bb7-aea2-23269ded877d-kube-api-access-fr6wr\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711576 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711509    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711576 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711524    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54f8ef22-0f82-4bb7-aea2-23269ded877d-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711576 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711537    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3277d75c-b24f-4a79-987e-a2128749a7c7-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711576 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711551    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh9pd\" (UniqueName: \"kubernetes.io/projected/3277d75c-b24f-4a79-987e-a2128749a7c7-kube-api-access-fh9pd\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711576 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711565    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5455c3c-e54d-4088-a012-688845de714a-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711794 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711580    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfq4\" (UniqueName: \"kubernetes.io/projected/b5455c3c-e54d-4088-a012-688845de714a-kube-api-access-dwfq4\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711794 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711594    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5455c3c-e54d-4088-a012-688845de714a-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711794 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711609    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711794 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711621    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3277d75c-b24f-4a79-987e-a2128749a7c7-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.711794 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.711634    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54f8ef22-0f82-4bb7-aea2-23269ded877d-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:59.905641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.905550    5718 generic.go:356] "Generic (PLEG): container finished" podID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerID="337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905" exitCode=0
Oct 31 02:08:59.905641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.905601    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"
Oct 31 02:08:59.905641 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.905639    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" event={"ID":"3277d75c-b24f-4a79-987e-a2128749a7c7","Type":"ContainerDied","Data":"337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905"}
Oct 31 02:08:59.905939 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.905704    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv" event={"ID":"3277d75c-b24f-4a79-987e-a2128749a7c7","Type":"ContainerDied","Data":"bf41cfc439eb9a776a2660c19c1a607b787e6224035ff70b9c094cccfb29a54b"}
Oct 31 02:08:59.905939 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.905724    5718 scope.go:122] "RemoveContainer" containerID="337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905"
Oct 31 02:08:59.908414 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.908232    5718 generic.go:356] "Generic (PLEG): container finished" podID="b5455c3c-e54d-4088-a012-688845de714a" containerID="138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c" exitCode=0
Oct 31 02:08:59.908414 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.908327    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" event={"ID":"b5455c3c-e54d-4088-a012-688845de714a","Type":"ContainerDied","Data":"138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c"}
Oct 31 02:08:59.908414 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.908363    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l" event={"ID":"b5455c3c-e54d-4088-a012-688845de714a","Type":"ContainerDied","Data":"7cb2df619a52b8a95fb84482c2a80a9178df9d2f315b9fba3646a4b4dffd78c9"}
Oct 31 02:08:59.908620 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.908434    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"
Oct 31 02:08:59.910786 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.910762    5718 generic.go:356] "Generic (PLEG): container finished" podID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerID="0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec" exitCode=0
Oct 31 02:08:59.910918 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.910829    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"
Oct 31 02:08:59.911188 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.910849    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" event={"ID":"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff","Type":"ContainerDied","Data":"0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec"}
Oct 31 02:08:59.911290 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.911199    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv" event={"ID":"5fa64d50-608d-4a3d-b5f8-c630ef9c23ff","Type":"ContainerDied","Data":"85590edd1ad5d7775fc2a6552aa68415fc4cbb548b810e242ad8686b5dcf128c"}
Oct 31 02:08:59.912427 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.912399    5718 generic.go:356] "Generic (PLEG): container finished" podID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerID="a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e" exitCode=0
Oct 31 02:08:59.913358 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.912461    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" event={"ID":"54f8ef22-0f82-4bb7-aea2-23269ded877d","Type":"ContainerDied","Data":"a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e"}
Oct 31 02:08:59.913358 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.912490    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f" event={"ID":"54f8ef22-0f82-4bb7-aea2-23269ded877d","Type":"ContainerDied","Data":"3bd7651d24edd55423decf8f1daddff47356ae5a119146edfadcc5613b7a2442"}
Oct 31 02:08:59.913358 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.912545    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"
Oct 31 02:08:59.928271 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.928232    5718 scope.go:122] "RemoveContainer" containerID="337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905"
Oct 31 02:08:59.928839 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:59.928811    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905\": container with ID starting with 337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905 not found: ID does not exist" containerID="337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905"
Oct 31 02:08:59.928972 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.928849    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905"} err="failed to get container status \"337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905\": rpc error: code = NotFound desc = could not find container \"337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905\": container with ID starting with 337a20f75dd52381307b4808b2fafd9e3d12495b6990ca1bcc8001f43a62b905 not found: ID does not exist"
Oct 31 02:08:59.928972 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.928872    5718 scope.go:122] "RemoveContainer" containerID="138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c"
Oct 31 02:08:59.939847 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.939826    5718 scope.go:122] "RemoveContainer" containerID="138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c"
Oct 31 02:08:59.940124 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:59.940105    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c\": container with ID starting with 138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c not found: ID does not exist" containerID="138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c"
Oct 31 02:08:59.940220 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.940132    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c"} err="failed to get container status \"138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c\": rpc error: code = NotFound desc = could not find container \"138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c\": container with ID starting with 138d7883fe98630821c1271a32e2a87be019caac01c62a1113923327765a1d1c not found: ID does not exist"
Oct 31 02:08:59.940220 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.940151    5718 scope.go:122] "RemoveContainer" containerID="0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec"
Oct 31 02:08:59.951142 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.951124    5718 scope.go:122] "RemoveContainer" containerID="0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec"
Oct 31 02:08:59.951464 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:59.951437    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec\": container with ID starting with 0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec not found: ID does not exist" containerID="0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec"
Oct 31 02:08:59.951585 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.951471    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec"} err="failed to get container status \"0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec\": rpc error: code = NotFound desc = could not find container \"0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec\": container with ID starting with 0cda7e060d2e1639e0c11caf76dba35aebcad70ca520f9e8035ba8631a3195ec not found: ID does not exist"
Oct 31 02:08:59.951585 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.951489    5718 scope.go:122] "RemoveContainer" containerID="a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e"
Oct 31 02:08:59.961238 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.961212    5718 scope.go:122] "RemoveContainer" containerID="a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e"
Oct 31 02:08:59.961604 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:08:59.961576    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e\": container with ID starting with a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e not found: ID does not exist" containerID="a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e"
Oct 31 02:08:59.961725 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:08:59.961616    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e"} err="failed to get container status \"a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e\": rpc error: code = NotFound desc = could not find container \"a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e\": container with ID starting with a4b7119e7bd96a560e338228de25673ee5dc1c81066daee2bf6d26ba422e0d8e not found: ID does not exist"
Oct 31 02:09:00.072389 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.072345    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"]
Oct 31 02:09:00.074336 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.074308    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/premium-w0-057e2d-kserve-6bcfbd54d9-6d92f"]
Oct 31 02:09:00.183986 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.183901    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"]
Oct 31 02:09:00.187678 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.187626    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/simulator-w0-057e2d-kserve-65d79c7564-pj96l"]
Oct 31 02:09:00.298704 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.298668    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"]
Oct 31 02:09:00.304170 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.304143    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-w0-057e2d-kserve-6b4cb687b8-zcvbv"]
Oct 31 02:09:00.413864 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.413830    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"]
Oct 31 02:09:00.418934 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:00.418907    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057e2d/distinct-2-w0-057e2d-kserve-758d479ff5-zgpxv"]
Oct 31 02:09:01.966351 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:01.966307    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" path="/var/lib/kubelet/pods/3277d75c-b24f-4a79-987e-a2128749a7c7/volumes"
Oct 31 02:09:01.966976 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:01.966955    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" path="/var/lib/kubelet/pods/54f8ef22-0f82-4bb7-aea2-23269ded877d/volumes"
Oct 31 02:09:01.967402 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:01.967383    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" path="/var/lib/kubelet/pods/5fa64d50-608d-4a3d-b5f8-c630ef9c23ff/volumes"
Oct 31 02:09:01.968152 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:01.968130    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5455c3c-e54d-4088-a012-688845de714a" path="/var/lib/kubelet/pods/b5455c3c-e54d-4088-a012-688845de714a/volumes"
Oct 31 02:09:02.972434 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:02.972398    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"]
Oct 31 02:09:07.925038 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:07.924999    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:09:10.541399 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:10.541364    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"]
Oct 31 02:09:10.646790 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:10.646749    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"]
Oct 31 02:09:15.542432 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.542392    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q" podUID="c14ce155-9a74-4b39-a491-9ea3b2270079" containerName="payload-processing" containerID="cri-o://f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d" gracePeriod=25
Oct 31 02:09:15.647124 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.647081    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk" podUID="f4430ffe-4178-4d65-903d-9f7ff0b74391" containerName="payload-pre-processing" containerID="cri-o://8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743" gracePeriod=25
Oct 31 02:09:15.805051 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.804992    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:09:15.887823 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.887792    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9\" (UniqueName: \"kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9\") pod \"c14ce155-9a74-4b39-a491-9ea3b2270079\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") "
Oct 31 02:09:15.888004 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.887982    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume\") pod \"c14ce155-9a74-4b39-a491-9ea3b2270079\" (UID: \"c14ce155-9a74-4b39-a491-9ea3b2270079\") "
Oct 31 02:09:15.888348 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.888325    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume" pod "c14ce155-9a74-4b39-a491-9ea3b2270079" (UID: "c14ce155-9a74-4b39-a491-9ea3b2270079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:15.890131 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.890107    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9" pod "c14ce155-9a74-4b39-a491-9ea3b2270079" (UID: "c14ce155-9a74-4b39-a491-9ea3b2270079"). InnerVolumeSpecName "kube-api-access-pckl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:15.909294 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.909268    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:09:15.989335 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.989305    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc\" (UniqueName: \"kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc\") pod \"f4430ffe-4178-4d65-903d-9f7ff0b74391\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") "
Oct 31 02:09:15.989475 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.989404    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume\") pod \"f4430ffe-4178-4d65-903d-9f7ff0b74391\" (UID: \"f4430ffe-4178-4d65-903d-9f7ff0b74391\") "
Oct 31 02:09:15.989789 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.989770    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pckl9\" (UniqueName: \"kubernetes.io/projected/c14ce155-9a74-4b39-a491-9ea3b2270079-kube-api-access-pckl9\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:15.989841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.989798    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14ce155-9a74-4b39-a491-9ea3b2270079-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:15.989841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.989816    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume" pod "f4430ffe-4178-4d65-903d-9f7ff0b74391" (UID: "f4430ffe-4178-4d65-903d-9f7ff0b74391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:15.991457 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:15.991434    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc" pod "f4430ffe-4178-4d65-903d-9f7ff0b74391" (UID: "f4430ffe-4178-4d65-903d-9f7ff0b74391"). InnerVolumeSpecName "kube-api-access-gl5zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:16.013149 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.013120    5718 generic.go:356] "Generic (PLEG): container finished" podID="c14ce155-9a74-4b39-a491-9ea3b2270079" containerID="f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d" exitCode=0
Oct 31 02:09:16.013263 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.013174    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"
Oct 31 02:09:16.013263 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.013205    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q" event={"ID":"c14ce155-9a74-4b39-a491-9ea3b2270079","Type":"ContainerDied","Data":"f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d"}
Oct 31 02:09:16.013263 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.013259    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q" event={"ID":"c14ce155-9a74-4b39-a491-9ea3b2270079","Type":"ContainerDied","Data":"41471e9d900467bfbaf215677da3c8a8b6283fafc50a380fa00543d4d33bf946"}
Oct 31 02:09:16.013419 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.013288    5718 scope.go:122] "RemoveContainer" containerID="f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d"
Oct 31 02:09:16.014723 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.014700    5718 generic.go:356] "Generic (PLEG): container finished" podID="f4430ffe-4178-4d65-903d-9f7ff0b74391" containerID="8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743" exitCode=0
Oct 31 02:09:16.014826 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.014761    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"
Oct 31 02:09:16.014826 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.014790    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk" event={"ID":"f4430ffe-4178-4d65-903d-9f7ff0b74391","Type":"ContainerDied","Data":"8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743"}
Oct 31 02:09:16.014908 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.014832    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk" event={"ID":"f4430ffe-4178-4d65-903d-9f7ff0b74391","Type":"ContainerDied","Data":"2b424633babbefe57281031d1a770053edad6d46dad5322c56c4169bd7e86fa6"}
Oct 31 02:09:16.024191 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.024170    5718 scope.go:122] "RemoveContainer" containerID="f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d"
Oct 31 02:09:16.024479 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:16.024459    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d\": container with ID starting with f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d not found: ID does not exist" containerID="f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d"
Oct 31 02:09:16.024548 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.024490    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d"} err="failed to get container status \"f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d\": rpc error: code = NotFound desc = could not find container \"f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d\": container with ID starting with f9707289e16eb535d49c1f822921d7debf98dbdbd860fa472a97fa0ba3da6b7d not found: ID does not exist"
Oct 31 02:09:16.024548 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.024510    5718 scope.go:122] "RemoveContainer" containerID="8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743"
Oct 31 02:09:16.032720 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.032700    5718 scope.go:122] "RemoveContainer" containerID="8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743"
Oct 31 02:09:16.033009 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:16.032989    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743\": container with ID starting with 8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743 not found: ID does not exist" containerID="8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743"
Oct 31 02:09:16.033057 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.033016    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743"} err="failed to get container status \"8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743\": rpc error: code = NotFound desc = could not find container \"8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743\": container with ID starting with 8e5939f5047552c3ad22076352e8326a0b499696c5ed9c9dc05e65e1c9016743 not found: ID does not exist"
Oct 31 02:09:16.090588 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.090506    5718 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4430ffe-4178-4d65-903d-9f7ff0b74391-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.090588 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.090536    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl5zc\" (UniqueName: \"kubernetes.io/projected/f4430ffe-4178-4d65-903d-9f7ff0b74391-kube-api-access-gl5zc\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.149966 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.149929    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"]
Oct 31 02:09:16.155966 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.155939    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-183266-6b84795bc-lpxpk"]
Oct 31 02:09:16.265287 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.265241    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"]
Oct 31 02:09:16.267862 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.267835    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-183266-79fdd5fc69-b7g4q"]
Oct 31 02:09:16.671481 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:16.671449    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"]
Oct 31 02:09:17.966346 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:17.966313    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14ce155-9a74-4b39-a491-9ea3b2270079" path="/var/lib/kubelet/pods/c14ce155-9a74-4b39-a491-9ea3b2270079/volumes"
Oct 31 02:09:17.966771 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:17.966752    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4430ffe-4178-4d65-903d-9f7ff0b74391" path="/var/lib/kubelet/pods/f4430ffe-4178-4d65-903d-9f7ff0b74391/volumes"
Oct 31 02:09:17.994442 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:17.994394    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq" podUID="46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" containerName="main" containerID="cri-o://f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556" gracePeriod=45
Oct 31 02:09:18.260968 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.260941    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:09:18.314150 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314117    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k\" (UniqueName: \"kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k\") pod \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") "
Oct 31 02:09:18.314373 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314176    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir\") pod \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") "
Oct 31 02:09:18.314373 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314290    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home\") pod \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") "
Oct 31 02:09:18.314373 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314329    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs\" (UniqueName: \"kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs\") pod \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") "
Oct 31 02:09:18.314543 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314382    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm\") pod \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") "
Oct 31 02:09:18.314543 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314432    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir" pod "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" (UID: "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:18.314543 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314512    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache\") pod \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\" (UID: \"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5\") "
Oct 31 02:09:18.314735 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314547    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home" pod "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" (UID: "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:18.314812 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314785    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache" pod "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" (UID: "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:18.314929 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314915    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:18.314969 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314933    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:18.314969 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.314943    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:18.316743 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.316712    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs" pod "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" (UID: "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:18.316880 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.316753    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm" pod "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" (UID: "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:18.316880 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.316787    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k" pod "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" (UID: "46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5"). InnerVolumeSpecName "kube-api-access-45p9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:18.416310 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.416271    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45p9k\" (UniqueName: \"kubernetes.io/projected/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-kube-api-access-45p9k\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:18.416310 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.416307    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:18.416516 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:18.416320    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.034546 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.034512    5718 generic.go:356] "Generic (PLEG): container finished" podID="46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" containerID="f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556" exitCode=0
Oct 31 02:09:19.035051 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.034555    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"
Oct 31 02:09:19.035051 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.034589    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq" event={"ID":"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5","Type":"ContainerDied","Data":"f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556"}
Oct 31 02:09:19.035051 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.034631    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq" event={"ID":"46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5","Type":"ContainerDied","Data":"4fd5de5e4d0ec422688ceb38c6a5c075d7a14176a8a5b2acc12232ea413db497"}
Oct 31 02:09:19.035051 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.034672    5718 scope.go:122] "RemoveContainer" containerID="f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556"
Oct 31 02:09:19.044766 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.044744    5718 scope.go:122] "RemoveContainer" containerID="f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556"
Oct 31 02:09:19.045078 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:19.045059    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556\": container with ID starting with f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556 not found: ID does not exist" containerID="f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556"
Oct 31 02:09:19.045142 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.045088    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556"} err="failed to get container status \"f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556\": rpc error: code = NotFound desc = could not find container \"f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556\": container with ID starting with f11bf9e3eceec485b5708ace4436f57954099ec907de0352604f4d14c58ff556 not found: ID does not exist"
Oct 31 02:09:19.199709 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.199679    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"]
Oct 31 02:09:19.249804 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.249763    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5632ea/auth-test-model-5632ea-kserve-586585f977-r9jzq"]
Oct 31 02:09:19.966020 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:19.965989    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" path="/var/lib/kubelet/pods/46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5/volumes"
Oct 31 02:09:23.442747 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.442666    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"]
Oct 31 02:09:23.443128 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.442870    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m" podUID="f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" containerName="maas-api" containerID="cri-o://b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba" gracePeriod=30
Oct 31 02:09:23.724920 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.724892    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:09:23.767210 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.767172    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca\") pod \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") "
Oct 31 02:09:23.767369 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.767226    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs\") pod \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") "
Oct 31 02:09:23.767369 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.767299    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt\" (UniqueName: \"kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt\") pod \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") "
Oct 31 02:09:23.767369 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.767362    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\") pod \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\" (UID: \"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0\") "
Oct 31 02:09:23.767682 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.767622    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca" pod "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" (UID: "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:23.769930 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.769739    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt" pod "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" (UID: "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0"). InnerVolumeSpecName "kube-api-access-h4dpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:23.770222 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.770189    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls" pod "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" (UID: "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:23.770222 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.770211    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs" pod "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" (UID: "f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:23.868480 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.868440    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dpt\" (UniqueName: \"kubernetes.io/projected/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-kube-api-access-h4dpt\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:23.868480 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.868477    5718 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-maas-api-tls\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:23.868480 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.868488    5718 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-openshift-service-ca\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:23.868838 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:23.868497    5718 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0-metrics-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:24.070816 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.070775    5718 generic.go:356] "Generic (PLEG): container finished" podID="f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" containerID="b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba" exitCode=0
Oct 31 02:09:24.071024 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.070832    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"
Oct 31 02:09:24.071024 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.070853    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m" event={"ID":"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0","Type":"ContainerDied","Data":"b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba"}
Oct 31 02:09:24.071024 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.070925    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m" event={"ID":"f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0","Type":"ContainerDied","Data":"06862ec187d0aa8380db21cd6ed0ad611bf2e4815e6ff91fba234bfc2e2aee78"}
Oct 31 02:09:24.071024 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.070958    5718 scope.go:122] "RemoveContainer" containerID="b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba"
Oct 31 02:09:24.084397 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.084368    5718 scope.go:122] "RemoveContainer" containerID="b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba"
Oct 31 02:09:24.085181 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:24.085152    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba\": container with ID starting with b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba not found: ID does not exist" containerID="b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba"
Oct 31 02:09:24.085275 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.085190    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba"} err="failed to get container status \"b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba\": rpc error: code = NotFound desc = could not find container \"b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba\": container with ID starting with b4a1bf95d540fc2252dea1afb7ef9d45aa961eec92a062d10b57dbc45a3d03ba not found: ID does not exist"
Oct 31 02:09:24.236732 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.236622    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"]
Oct 31 02:09:24.238134 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:24.238108    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-183266-bd4475b97-hq64m"]
Oct 31 02:09:25.967531 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:25.967494    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" path="/var/lib/kubelet/pods/f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0/volumes"
Oct 31 02:09:31.692748 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:31.692705    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main" containerID="cri-o://6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59" gracePeriod=45
Oct 31 02:09:31.778240 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:31.778203    5718 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused"
Oct 31 02:09:31.952012 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:31.951945    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 02:09:32.048094 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048058    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home\") pod \"815614a9-1038-4ad5-aed5-057bece8b655\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") "
Oct 31 02:09:32.048094 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048095    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir\") pod \"815614a9-1038-4ad5-aed5-057bece8b655\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") "
Oct 31 02:09:32.048344 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048124    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache\") pod \"815614a9-1038-4ad5-aed5-057bece8b655\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") "
Oct 31 02:09:32.048344 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048201    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs\" (UniqueName: \"kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs\") pod \"815614a9-1038-4ad5-aed5-057bece8b655\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") "
Oct 31 02:09:32.048344 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048304    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j\" (UniqueName: \"kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j\") pod \"815614a9-1038-4ad5-aed5-057bece8b655\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") "
Oct 31 02:09:32.048455 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048350    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home" pod "815614a9-1038-4ad5-aed5-057bece8b655" (UID: "815614a9-1038-4ad5-aed5-057bece8b655"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.048455 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048371    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm\") pod \"815614a9-1038-4ad5-aed5-057bece8b655\" (UID: \"815614a9-1038-4ad5-aed5-057bece8b655\") "
Oct 31 02:09:32.048455 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048384    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir" pod "815614a9-1038-4ad5-aed5-057bece8b655" (UID: "815614a9-1038-4ad5-aed5-057bece8b655"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.048568 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048466    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache" pod "815614a9-1038-4ad5-aed5-057bece8b655" (UID: "815614a9-1038-4ad5-aed5-057bece8b655"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.048822 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048803    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.048822 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048822    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.048822 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.048831    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.050889 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.050856    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j" pod "815614a9-1038-4ad5-aed5-057bece8b655" (UID: "815614a9-1038-4ad5-aed5-057bece8b655"). InnerVolumeSpecName "kube-api-access-8kq7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:32.051212 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.051186    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs" pod "815614a9-1038-4ad5-aed5-057bece8b655" (UID: "815614a9-1038-4ad5-aed5-057bece8b655"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:32.051643 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.051623    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm" pod "815614a9-1038-4ad5-aed5-057bece8b655" (UID: "815614a9-1038-4ad5-aed5-057bece8b655"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.120773 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.120728    5718 generic.go:356] "Generic (PLEG): container finished" podID="815614a9-1038-4ad5-aed5-057bece8b655" containerID="6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59" exitCode=0
Oct 31 02:09:32.121019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.120792    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"
Oct 31 02:09:32.121019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.120791    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" event={"ID":"815614a9-1038-4ad5-aed5-057bece8b655","Type":"ContainerDied","Data":"6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59"}
Oct 31 02:09:32.121019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.120859    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p" event={"ID":"815614a9-1038-4ad5-aed5-057bece8b655","Type":"ContainerDied","Data":"a9c79691fdcb10e614329c02724d8cd68f118c4429e684251fc5785552efcee9"}
Oct 31 02:09:32.121019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.120883    5718 scope.go:122] "RemoveContainer" containerID="6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59"
Oct 31 02:09:32.134422 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.134398    5718 scope.go:122] "RemoveContainer" containerID="6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59"
Oct 31 02:09:32.134798 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:32.134762    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59\": container with ID starting with 6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59 not found: ID does not exist" containerID="6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59"
Oct 31 02:09:32.134926 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.134806    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59"} err="failed to get container status \"6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59\": rpc error: code = NotFound desc = could not find container \"6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59\": container with ID starting with 6982d977549971f63c1d4d9af6d6b3f54733c9052d6ab50dc0f5aad8f86d6a59 not found: ID does not exist"
Oct 31 02:09:32.150342 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.150314    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815614a9-1038-4ad5-aed5-057bece8b655-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.150342 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.150340    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kq7j\" (UniqueName: \"kubernetes.io/projected/815614a9-1038-4ad5-aed5-057bece8b655-kube-api-access-8kq7j\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.150586 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.150352    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815614a9-1038-4ad5-aed5-057bece8b655-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.287023 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.286987    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"]
Oct 31 02:09:32.290945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:32.290919    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/premium-w3-a8e157-kserve-7f9778d54-xx79p"]
Oct 31 02:09:33.967931 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:09:33.967899    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815614a9-1038-4ad5-aed5-057bece8b655" path="/var/lib/kubelet/pods/815614a9-1038-4ad5-aed5-057bece8b655/volumes"
Oct 31 02:09:37.924454 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:09:37.924417    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:10:05.881605 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:05.881568    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"]
Oct 31 02:10:05.882158 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:05.881953    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy" containerID="cri-o://3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c" gracePeriod=30
Oct 31 02:10:07.923866 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:10:07.923830    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:10:11.137628 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.137600    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 02:10:11.272031 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.271995    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272230 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272058    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272230 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272114    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272230 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272176    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272420 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272271    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272420 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272314    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272420 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272347    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272420 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272390    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272420 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272416    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data\") pod \"e28711de-ee19-40ee-9e29-5049758bfd6e\" (UID: \"e28711de-ee19-40ee-9e29-5049758bfd6e\") "
Oct 31 02:10:11.272719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272425    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:11.272719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272459    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:11.272719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272464    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:11.272719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272485    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:11.272945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272823    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:11.272945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272833    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28711de-ee19-40ee-9e29-5049758bfd6e-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.272945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272858    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.272945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272873    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.272945 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.272889    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.274728 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.274706    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:11.274878 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.274790    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "kube-api-access-x88jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:11.275018 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.275002    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:11.275089 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.275072    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo" pod "e28711de-ee19-40ee-9e29-5049758bfd6e" (UID: "e28711de-ee19-40ee-9e29-5049758bfd6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:11.334906 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.334868    5718 generic.go:356] "Generic (PLEG): container finished" podID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerID="3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c" exitCode=0
Oct 31 02:10:11.335058 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.334919    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"
Oct 31 02:10:11.335058 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.334944    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" event={"ID":"e28711de-ee19-40ee-9e29-5049758bfd6e","Type":"ContainerDied","Data":"3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c"}
Oct 31 02:10:11.335058 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.334989    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw" event={"ID":"e28711de-ee19-40ee-9e29-5049758bfd6e","Type":"ContainerDied","Data":"fb35c8135c167d73a89150ca182bb76cbd655878ab8d4896f5386e4ffe5904f6"}
Oct 31 02:10:11.335058 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.335011    5718 scope.go:122] "RemoveContainer" containerID="3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c"
Oct 31 02:10:11.346890 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.346870    5718 scope.go:122] "RemoveContainer" containerID="3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c"
Oct 31 02:10:11.347182 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:10:11.347159    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c\": container with ID starting with 3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c not found: ID does not exist" containerID="3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c"
Oct 31 02:10:11.347311 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.347191    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c"} err="failed to get container status \"3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c\": rpc error: code = NotFound desc = could not find container \"3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c\": container with ID starting with 3dfaf885d4f7d6ad15d7cff2b432ea622de8f0eaac359dd6b154b3caf16da52c not found: ID does not exist"
Oct 31 02:10:11.373971 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.373940    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.373971 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.373969    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.374168 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.373982    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x88jm\" (UniqueName: \"kubernetes.io/projected/e28711de-ee19-40ee-9e29-5049758bfd6e-kube-api-access-x88jm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.374168 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.373998    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.374168 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.374009    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28711de-ee19-40ee-9e29-5049758bfd6e-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.499988 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.499951    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"]
Oct 31 02:10:11.503904 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.503874    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hnqlw"]
Oct 31 02:10:11.966219 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:10:11.966183    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" path="/var/lib/kubelet/pods/e28711de-ee19-40ee-9e29-5049758bfd6e/volumes"
Oct 31 02:10:37.923532 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:10:37.923488    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:11:01.786950 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:01.786868    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"]
Oct 31 02:11:01.837248 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:01.837210    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"]
Oct 31 02:11:06.918346 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:06.918307    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"]
Oct 31 02:11:06.918820 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:06.918548    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy" containerID="cri-o://295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da" gracePeriod=30
Oct 31 02:11:07.923412 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:11:07.923370    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:11:12.169028 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.169001    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 02:11:12.300758 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300722    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.300944 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300787    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.300944 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300846    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.300944 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300872    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.300944 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300929    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.301164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300959    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.301164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.300989    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.301164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301018    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.301164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301079    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert\") pod \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\" (UID: \"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8\") "
Oct 31 02:11:12.301391 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301199    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:12.301452 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301406    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:12.301509 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301474    5718 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-data\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.301572 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301500    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:12.301572 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301523    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:12.301801 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.301686    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:12.303313 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.303290    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "kube-api-access-f5vb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:12.303471 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.303457    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:11:12.303555 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.303542    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:12.303602 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.303582    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token" pod "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" (UID: "d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:12.402582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402544    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vb4\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-kube-api-access-f5vb4\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402577    5718 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-podinfo\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402582 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402589    5718 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402860 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402597    5718 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-credential-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402860 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402606    5718 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-workload-socket\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402860 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402615    5718 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402860 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402623    5718 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-token\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.402860 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.402631    5718 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8-istio-envoy\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.647293 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.647206    5718 generic.go:356] "Generic (PLEG): container finished" podID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerID="295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da" exitCode=0
Oct 31 02:11:12.647293 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.647255    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"
Oct 31 02:11:12.647293 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.647274    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" event={"ID":"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8","Type":"ContainerDied","Data":"295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da"}
Oct 31 02:11:12.647578 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.647337    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb" event={"ID":"d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8","Type":"ContainerDied","Data":"275399dab0f4b51383e8e7bd87a9d260278982bccf70631203a1f41bcd4f1371"}
Oct 31 02:11:12.647578 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.647366    5718 scope.go:122] "RemoveContainer" containerID="295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da"
Oct 31 02:11:12.658460 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.658434    5718 scope.go:122] "RemoveContainer" containerID="295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da"
Oct 31 02:11:12.658826 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:11:12.658804    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da\": container with ID starting with 295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da not found: ID does not exist" containerID="295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da"
Oct 31 02:11:12.658917 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.658836    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da"} err="failed to get container status \"295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da\": rpc error: code = NotFound desc = could not find container \"295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da\": container with ID starting with 295cba76ce788ccbfc220516d532a3448fb80dda29dd6df412117a5d73aa51da not found: ID does not exist"
Oct 31 02:11:12.812910 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.812864    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"]
Oct 31 02:11:12.863762 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:12.863726    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-csgqb"]
Oct 31 02:11:13.966840 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:13.966806    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" path="/var/lib/kubelet/pods/d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8/volumes"
Oct 31 02:11:16.808580 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:16.808532    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main" containerID="cri-o://25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f" gracePeriod=45
Oct 31 02:11:16.857972 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:16.857929    5718 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main" containerID="cri-o://d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2" gracePeriod=45
Oct 31 02:11:17.115311 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.115286    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 02:11:17.118274 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.118252    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 02:11:17.250182 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250151    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v\" (UniqueName: \"kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v\") pod \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") "
Oct 31 02:11:17.250387 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250220    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache\") pod \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") "
Oct 31 02:11:17.250387 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250325    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home\") pod \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") "
Oct 31 02:11:17.250387 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250353    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home\") pod \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250387    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir\") pod \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250412    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz\" (UniqueName: \"kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz\") pod \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250443    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir\") pod \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250477    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm\") pod \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250506    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm\") pod \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250535    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs\" (UniqueName: \"kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs\") pod \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") "
Oct 31 02:11:17.250565 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250548    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache" pod "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" (UID: "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.250925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250571    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs\") pod \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\" (UID: \"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d\") "
Oct 31 02:11:17.250925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250560    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home" pod "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" (UID: "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.250925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250619    5718 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache\") pod \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\" (UID: \"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5\") "
Oct 31 02:11:17.250925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250814    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir" pod "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" (UID: "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.250925 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.250822    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir" pod "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" (UID: "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.251195 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.251055    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache" pod "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" (UID: "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.251195 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.251079    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.251195 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.251102    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.251195 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.251117    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.251195 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.251130    5718 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tmp-dir\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.251470 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.251195    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home" pod "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" (UID: "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.253604 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.253575    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs" pod "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" (UID: "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:17.253783 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.253618    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz" pod "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" (UID: "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d"). InnerVolumeSpecName "kube-api-access-89rpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:17.253783 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.253748    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v" pod "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" (UID: "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5"). InnerVolumeSpecName "kube-api-access-p7m4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:17.253974 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.253953    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm" pod "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" (UID: "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.254054 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.253969    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs" pod "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" (UID: "36cded3f-8e56-4db3-b3e9-48f9b1bd7df5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:17.255062 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.255038    5718 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm" pod "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" (UID: "bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.351926    5718 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-home\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.351958    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89rpz\" (UniqueName: \"kubernetes.io/projected/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-kube-api-access-89rpz\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.351969    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.351978    5718 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-dshm\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.351987    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.351999    5718 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d-tls-certs\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.352010    5718 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-model-cache\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.352019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.352024    5718 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7m4v\" (UniqueName: \"kubernetes.io/projected/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5-kube-api-access-p7m4v\") on node \"ip-10-0-141-86.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.675608 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.675529    5718 generic.go:356] "Generic (PLEG): container finished" podID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerID="25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f" exitCode=0
Oct 31 02:11:17.675608 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.675579    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"
Oct 31 02:11:17.675887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.675621    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" event={"ID":"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d","Type":"ContainerDied","Data":"25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f"}
Oct 31 02:11:17.675887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.675699    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q" event={"ID":"bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d","Type":"ContainerDied","Data":"73809208da53cf40c6a5231df176a35fb2ca5eeaac21b38082c535d9a5b81d1a"}
Oct 31 02:11:17.675887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.675722    5718 scope.go:122] "RemoveContainer" containerID="25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f"
Oct 31 02:11:17.677554 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.677528    5718 generic.go:356] "Generic (PLEG): container finished" podID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerID="d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2" exitCode=0
Oct 31 02:11:17.677713 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.677573    5718 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"
Oct 31 02:11:17.677713 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.677621    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" event={"ID":"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5","Type":"ContainerDied","Data":"d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2"}
Oct 31 02:11:17.677713 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.677685    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n" event={"ID":"36cded3f-8e56-4db3-b3e9-48f9b1bd7df5","Type":"ContainerDied","Data":"6a045853d5e42f605b1eb92c4a98c4a5903e7694883e33cffabccebe69f84eb6"}
Oct 31 02:11:17.686314 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.686199    5718 scope.go:122] "RemoveContainer" containerID="25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f"
Oct 31 02:11:17.686561 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:11:17.686539    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f\": container with ID starting with 25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f not found: ID does not exist" containerID="25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f"
Oct 31 02:11:17.686637 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.686570    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f"} err="failed to get container status \"25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f\": rpc error: code = NotFound desc = could not find container \"25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f\": container with ID starting with 25717ea6eeed334e98fe7685d571292050e65849a5c0ddf926d1fe3ebcc8448f not found: ID does not exist"
Oct 31 02:11:17.686637 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.686588    5718 scope.go:122] "RemoveContainer" containerID="d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2"
Oct 31 02:11:17.698469 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.698448    5718 scope.go:122] "RemoveContainer" containerID="d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2"
Oct 31 02:11:17.698818 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:11:17.698796    5718 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2\": container with ID starting with d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2 not found: ID does not exist" containerID="d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2"
Oct 31 02:11:17.698881 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.698828    5718 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2"} err="failed to get container status \"d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2\": rpc error: code = NotFound desc = could not find container \"d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2\": container with ID starting with d7c7657b385169e36366259769dfa4fd992ad10389c0baa76028f54b070032d2 not found: ID does not exist"
Oct 31 02:11:17.840054 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.840021    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"]
Oct 31 02:11:17.890566 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.890527    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/simulator-w1-baacee-kserve-76cf886d9c-6n59n"]
Oct 31 02:11:17.967232 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.967143    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" path="/var/lib/kubelet/pods/36cded3f-8e56-4db3-b3e9-48f9b1bd7df5/volumes"
Oct 31 02:11:17.999831 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:17.999799    5718 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"]
Oct 31 02:11:18.003164 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:18.003137    5718 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-2-w1-baacee-kserve-78f5bb5c96-mct6q"]
Oct 31 02:11:19.966803 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:19.966765    5718 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" path="/var/lib/kubelet/pods/bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d/volumes"
Oct 31 02:11:35.628324 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:35.628297    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m5wd8_d95ec0ac-a995-42ea-acfa-d3566d1d742b/kuadrant-console-plugin/0.log"
Oct 31 02:11:37.922905 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:11:37.922866    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:11:38.474452 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:38.474419    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:11:38.781186 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:38.781142    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:11:39.108362 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:39.108277    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:11:39.735859 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:39.735825    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:11:39.988372 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:39.988287    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:11:40.545863 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:40.545831    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:11:52.536182 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:52.536141    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:11:52.832381 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:52.832296    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:11:53.152174 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:53.152088    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:11:53.728094 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:53.728058    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:11:53.969995 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:53.969962    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:11:54.633289 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:54.633256    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:11:55.481108 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:55.481078    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84cdb66567-djg7q_1a902ea0-8131-43d8-a900-96da008d21e2/maas-api/0.log"
Oct 31 02:11:58.508882 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:11:58.508852    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-zfzh7_2388db30-e28c-4c77-85d3-6f9e83e09eeb/postgres/0.log"
Oct 31 02:12:00.989401 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:12:00.989371    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m5wd8_d95ec0ac-a995-42ea-acfa-d3566d1d742b/kuadrant-console-plugin/0.log"
Oct 31 02:12:07.923590 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:12:07.923549    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:12:24.184604 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:12:24.184507    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-j96vj_6aba0896-e4fb-4b1d-9904-ca4e55c59e82/kube-auth-proxy/0.log"
Oct 31 02:12:24.701130 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:12:24.701101    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9mzcr_39645c3b-b2f4-448a-8683-c7b5b4ec1e51/istio-proxy/0.log"
Oct 31 02:12:25.052006 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:12:25.051975    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks_96a77a21-5cc4-4025-bd5c-4c2492aac124/istio-proxy/0.log"
Oct 31 02:12:37.923227 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:12:37.923184    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:12:58.047832 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:12:58.047799    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pf5kd_2e497701-d9b9-46b7-a5db-db294421d12f/global-pull-secret-syncer/0.log"
Oct 31 02:13:00.552575 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:00.552542    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:00.812444 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:00.812353    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:01.319797 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:01.319764    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:01.613887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:01.613794    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:01.675098 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:01.675061    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:02.028740 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:02.028704    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j6h6v_caa76e03-b876-4bc5-a0ec-aab3cced55f0/konnectivity-agent/0.log"
Oct 31 02:13:02.230672 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:02.230628    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:02.341070 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:02.340977    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:02.455374 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:02.455324    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:02.878626 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:02.878586    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:02.989358 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:02.989329    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:03.209782 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:03.209695    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-86.us-west-1.compute.internal_fafe7c61d6af085c60e65c5248ab0754/haproxy/0.log"
Oct 31 02:13:03.551701 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:03.551667    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:04.202823 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:04.202791    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:06.773096 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:06.773063    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:07.035011 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:07.034918    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:07.224473 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:07.224438    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:07.520633 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:07.520599    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:07.931248 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:13:07.931151    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:13:08.035887 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:08.035825    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:08.441454 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:08.441419    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:08.561194 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:08.561146    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:08.686468 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:08.686439    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:08.860692 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:08.860632    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:09.011866 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:09.011838    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:11.581343 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:11.581312    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:11.926088 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:11.926009    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:14.973715 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:14.973685    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:15.127719 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:15.127670    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:15.235507 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:15.235402    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:15.518767 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:15.518738    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:16.232225 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:16.232181    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:16.432019 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:16.431982    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:16.739111 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:16.739068    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:16.857030 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:16.856995    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:16.913895 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:16.913854    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:17.041493 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:17.041459    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:18.552698 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:18.552642    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:18.667580 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:18.667550    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:19.734414 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:19.734383    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m5wd8_d95ec0ac-a995-42ea-acfa-d3566d1d742b/kuadrant-console-plugin/0.log"
Oct 31 02:13:22.751383 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:22.751349    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-j96vj_6aba0896-e4fb-4b1d-9904-ca4e55c59e82/kube-auth-proxy/0.log"
Oct 31 02:13:23.037111 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:23.037030    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-j96vj_6aba0896-e4fb-4b1d-9904-ca4e55c59e82/kube-auth-proxy/0.log"
Oct 31 02:13:23.185319 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:23.185286    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9mzcr_39645c3b-b2f4-448a-8683-c7b5b4ec1e51/istio-proxy/0.log"
Oct 31 02:13:23.393413 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:23.393319    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks_96a77a21-5cc4-4025-bd5c-4c2492aac124/istio-proxy/0.log"
Oct 31 02:13:23.466614 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:23.466581    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9mzcr_39645c3b-b2f4-448a-8683-c7b5b4ec1e51/istio-proxy/0.log"
Oct 31 02:13:23.682304 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:23.682223    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks_96a77a21-5cc4-4025-bd5c-4c2492aac124/istio-proxy/0.log"
Oct 31 02:13:27.571289 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:27.571252    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rk4z5_05380ea7-efbf-4c24-9163-97a4243365df/kube-state-metrics/0.log"
Oct 31 02:13:28.021768 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.021742    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rk4z5_05380ea7-efbf-4c24-9163-97a4243365df/kube-rbac-proxy-main/0.log"
Oct 31 02:13:28.274034 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.273955    5718 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"]
Oct 31 02:13:28.274353 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274341    5718 cpu_manager.go:414] "containerMap: removing container" podUID="f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" containerName="maas-api"
Oct 31 02:13:28.274398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274355    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" containerName="maas-api"
Oct 31 02:13:28.274398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274366    5718 cpu_manager.go:414] "containerMap: removing container" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerName="main"
Oct 31 02:13:28.274398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274371    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerName="main"
Oct 31 02:13:28.274398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274381    5718 cpu_manager.go:414] "containerMap: removing container" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy"
Oct 31 02:13:28.274398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274386    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy"
Oct 31 02:13:28.274398 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274397    5718 cpu_manager.go:414] "containerMap: removing container" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274402    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274420    5718 cpu_manager.go:414] "containerMap: removing container" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274425    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274433    5718 cpu_manager.go:414] "containerMap: removing container" podUID="d660f4e0-a3e1-4095-a42e-b2924b01071c" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274438    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d660f4e0-a3e1-4095-a42e-b2924b01071c" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274443    5718 cpu_manager.go:414] "containerMap: removing container" podUID="b91d1b08-9e21-4a52-a31d-9c7989db39b6" containerName="maas-api"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274447    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91d1b08-9e21-4a52-a31d-9c7989db39b6" containerName="maas-api"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274454    5718 cpu_manager.go:414] "containerMap: removing container" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274459    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274469    5718 cpu_manager.go:414] "containerMap: removing container" podUID="7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274474    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274479    5718 cpu_manager.go:414] "containerMap: removing container" podUID="f4430ffe-4178-4d65-903d-9f7ff0b74391" containerName="payload-pre-processing"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274484    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4430ffe-4178-4d65-903d-9f7ff0b74391" containerName="payload-pre-processing"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274493    5718 cpu_manager.go:414] "containerMap: removing container" podUID="b40c8b6a-e6f9-4248-9c10-ec57d60464b0" containerName="maas-api"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274497    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40c8b6a-e6f9-4248-9c10-ec57d60464b0" containerName="maas-api"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274504    5718 cpu_manager.go:414] "containerMap: removing container" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274508    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274515    5718 cpu_manager.go:414] "containerMap: removing container" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274520    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274525    5718 cpu_manager.go:414] "containerMap: removing container" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274529    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274536    5718 cpu_manager.go:414] "containerMap: removing container" podUID="145e629f-8852-4a5e-9286-5ad9e17eb96e" containerName="maas-api"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274540    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145e629f-8852-4a5e-9286-5ad9e17eb96e" containerName="maas-api"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274549    5718 cpu_manager.go:414] "containerMap: removing container" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274553    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274558    5718 cpu_manager.go:414] "containerMap: removing container" podUID="46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274562    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" containerName="main"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274567    5718 cpu_manager.go:414] "containerMap: removing container" podUID="c14ce155-9a74-4b39-a491-9ea3b2270079" containerName="payload-processing"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274572    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14ce155-9a74-4b39-a491-9ea3b2270079" containerName="payload-processing"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274579    5718 cpu_manager.go:414] "containerMap: removing container" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy"
Oct 31 02:13:28.274594 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274583    5718 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274637    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="3277d75c-b24f-4a79-987e-a2128749a7c7" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274646    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa64d50-608d-4a3d-b5f8-c630ef9c23ff" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274669    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28711de-ee19-40ee-9e29-5049758bfd6e" containerName="istio-proxy"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274677    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="d660f4e0-a3e1-4095-a42e-b2924b01071c" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274689    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cded3f-8e56-4db3-b3e9-48f9b1bd7df5" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274696    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="7771fbf3-2ed2-477e-890d-f9c2b7a6b5a9" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274705    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4430ffe-4178-4d65-903d-9f7ff0b74391" containerName="payload-pre-processing"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274716    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40c8b6a-e6f9-4248-9c10-ec57d60464b0" containerName="maas-api"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274722    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f8ef22-0f82-4bb7-aea2-23269ded877d" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274728    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="815614a9-1038-4ad5-aed5-057bece8b655" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274733    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43fa747-14b6-4aa0-8684-3cfd3f6fc1c8" containerName="istio-proxy"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274738    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="145e629f-8852-4a5e-9286-5ad9e17eb96e" containerName="maas-api"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274744    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e7fb86-a7f6-4fd4-ae78-7d6e879f4ff5" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274749    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9b7ddb-7122-4d82-8fd7-6e7148f29a8d" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274754    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14ce155-9a74-4b39-a491-9ea3b2270079" containerName="payload-processing"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274760    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5455c3c-e54d-4088-a012-688845de714a" containerName="main"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274765    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c783693-7a55-42cf-a4c6-983a680377e9" containerName="istio-proxy"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274771    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d94c9c-a085-4f7b-8022-a26c2b9ea8a0" containerName="maas-api"
Oct 31 02:13:28.275514 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.274776    5718 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91d1b08-9e21-4a52-a31d-9c7989db39b6" containerName="maas-api"
Oct 31 02:13:28.277817 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.277802    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.333692 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.333646    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"]
Oct 31 02:13:28.336163 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.335127    5718 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-frc7g\"/\"default-dockercfg-lp57d\""
Oct 31 02:13:28.336163 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.335309    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-frc7g\"/\"kube-root-ca.crt\""
Oct 31 02:13:28.370015 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.369977    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-podres\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.370175 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.370023    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-sys\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.370175 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.370044    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-lib-modules\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.370269 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.370187    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-proc\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.370269 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.370231    5718 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rv2z\" (UniqueName: \"kubernetes.io/projected/ee4712fd-5028-47ce-80af-2c8681680b16-kube-api-access-2rv2z\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.382577 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.382550    5718 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-frc7g\"/\"openshift-service-ca.crt\""
Oct 31 02:13:28.440368 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.440341    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-rk4z5_05380ea7-efbf-4c24-9163-97a4243365df/kube-rbac-proxy-self/0.log"
Oct 31 02:13:28.471571 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471542    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-proc\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471733 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471611    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rv2z\" (UniqueName: \"kubernetes.io/projected/ee4712fd-5028-47ce-80af-2c8681680b16-kube-api-access-2rv2z\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471733 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471677    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-proc\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471733 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471684    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-podres\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471899 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471738    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-podres\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471899 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471745    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-sys\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471899 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471780    5718 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-lib-modules\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471899 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471792    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-sys\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.471899 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.471850    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee4712fd-5028-47ce-80af-2c8681680b16-lib-modules\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.529272 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.529200    5718 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rv2z\" (UniqueName: \"kubernetes.io/projected/ee4712fd-5028-47ce-80af-2c8681680b16-kube-api-access-2rv2z\") pod \"perf-node-gather-daemonset-brh8g\" (UID: \"ee4712fd-5028-47ce-80af-2c8681680b16\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.589985 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.589952    5718 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:28.859436 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.859353    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-66f6f89fb8-2wbfh_432f223d-4785-4a5c-af6a-921a65b1a08e/metrics-server/0.log"
Oct 31 02:13:28.960487 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:28.960452    5718 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"]
Oct 31 02:13:28.961986 ip-10-0-141-86 kubenswrapper[5718]: W1031 02:13:28.961955    5718 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee4712fd_5028_47ce_80af_2c8681680b16.slice/crio-94ba66915c2ff10a351771059b864342b42b452d2665c88fb3e877fc8bdac946 WatchSource:0}: Error finding container 94ba66915c2ff10a351771059b864342b42b452d2665c88fb3e877fc8bdac946: Status 404 returned error can't find the container with id 94ba66915c2ff10a351771059b864342b42b452d2665c88fb3e877fc8bdac946
Oct 31 02:13:29.281033 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:29.281007    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-cc59x_120f0f3f-da9b-4395-807b-a291a37b5e2d/monitoring-plugin/0.log"
Oct 31 02:13:29.322935 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:29.322899    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g" event={"ID":"ee4712fd-5028-47ce-80af-2c8681680b16","Type":"ContainerStarted","Data":"cc56f3f8295ffe12046c99c4b31bb5d84726135d05518e7d17fe3e6748331c31"}
Oct 31 02:13:29.323128 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:29.322951    5718 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g" event={"ID":"ee4712fd-5028-47ce-80af-2c8681680b16","Type":"ContainerStarted","Data":"94ba66915c2ff10a351771059b864342b42b452d2665c88fb3e877fc8bdac946"}
Oct 31 02:13:29.323128 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:29.323025    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:29.793368 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:29.793341    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m5wd8_d95ec0ac-a995-42ea-acfa-d3566d1d742b/kuadrant-console-plugin/0.log"
Oct 31 02:13:30.210879 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:30.210808    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m5wd8_d95ec0ac-a995-42ea-acfa-d3566d1d742b/kuadrant-console-plugin/0.log"
Oct 31 02:13:32.277752 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:32.277719    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tm9rb_961e1a63-66d7-4c5f-924e-27d9d56d6862/node-exporter/0.log"
Oct 31 02:13:32.692827 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:32.692743    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tm9rb_961e1a63-66d7-4c5f-924e-27d9d56d6862/kube-rbac-proxy/0.log"
Oct 31 02:13:33.105434 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:33.105404    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tm9rb_961e1a63-66d7-4c5f-924e-27d9d56d6862/init-textfile/0.log"
Oct 31 02:13:33.575153 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:33.575106    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-h5ltp_65337868-194c-4e3f-9e3f-7e96c35c6761/kube-rbac-proxy-main/0.log"
Oct 31 02:13:34.035265 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:34.035219    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-h5ltp_65337868-194c-4e3f-9e3f-7e96c35c6761/kube-rbac-proxy-self/0.log"
Oct 31 02:13:34.497541 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:34.497508    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-h5ltp_65337868-194c-4e3f-9e3f-7e96c35c6761/openshift-state-metrics/0.log"
Oct 31 02:13:34.922147 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:34.922067    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/prometheus/0.log"
Oct 31 02:13:35.335286 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:35.335259    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/config-reloader/0.log"
Oct 31 02:13:35.339971 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:35.339945    5718 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g"
Oct 31 02:13:35.457034 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:35.456978    5718 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-brh8g" podStartSLOduration=7.456965341 podStartE2EDuration="7.456965341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:29.448148972 +0000 UTC m=+4125.611292387" watchObservedRunningTime="2026-10-31 02:13:35.456965341 +0000 UTC m=+4131.620108790"
Oct 31 02:13:35.749102 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:35.749054    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/thanos-sidecar/0.log"
Oct 31 02:13:36.164504 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:36.164417    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/kube-rbac-proxy-web/0.log"
Oct 31 02:13:36.579081 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:36.579048    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/kube-rbac-proxy/0.log"
Oct 31 02:13:36.993498 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:36.993423    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/kube-rbac-proxy-thanos/0.log"
Oct 31 02:13:37.410001 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:37.409973    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d90e5bc8-ff48-4edc-9a2e-16496cc7c497/init-config-reloader/0.log"
Oct 31 02:13:37.923495 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:13:37.923457    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:13:40.375486 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:40.375453    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f54b69d-l5mz6_911363c3-235f-4667-b12c-94089715f1bc/thanos-query/0.log"
Oct 31 02:13:40.789841 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:40.789811    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f54b69d-l5mz6_911363c3-235f-4667-b12c-94089715f1bc/kube-rbac-proxy-web/0.log"
Oct 31 02:13:41.206951 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:41.206863    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f54b69d-l5mz6_911363c3-235f-4667-b12c-94089715f1bc/kube-rbac-proxy/0.log"
Oct 31 02:13:41.621556 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:41.621526    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f54b69d-l5mz6_911363c3-235f-4667-b12c-94089715f1bc/prom-label-proxy/0.log"
Oct 31 02:13:42.037678 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:42.037631    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f54b69d-l5mz6_911363c3-235f-4667-b12c-94089715f1bc/kube-rbac-proxy-rules/0.log"
Oct 31 02:13:42.452881 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:42.452806    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f6f54b69d-l5mz6_911363c3-235f-4667-b12c-94089715f1bc/kube-rbac-proxy-metrics/0.log"
Oct 31 02:13:46.125834 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:46.125793    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-k64tr_0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407/cert-manager-controller/0.log"
Oct 31 02:13:46.542783 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:46.542755    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-qb6hx_54322f7b-53d0-4d84-be02-195f6296a5a7/cert-manager-webhook/0.log"
Oct 31 02:13:46.997606 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:46.997580    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-k64tr_0fe95ead-d8b9-4dcb-8b53-6ca36fa3b407/cert-manager-controller/0.log"
Oct 31 02:13:47.421977 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:47.421882    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-qb6hx_54322f7b-53d0-4d84-be02-195f6296a5a7/cert-manager-webhook/0.log"
Oct 31 02:13:48.767159 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:48.767120    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-2hltz_ee11ca63-ef7e-405e-bb4a-c44fcf8a5081/manager/0.log"
Oct 31 02:13:48.974103 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:48.974074    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-2hltz_ee11ca63-ef7e-405e-bb4a-c44fcf8a5081/copy-manifests/0.log"
Oct 31 02:13:49.653045 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:49.653014    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-2hltz_ee11ca63-ef7e-405e-bb4a-c44fcf8a5081/manager/0.log"
Oct 31 02:13:49.860453 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:49.860422    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-2hltz_ee11ca63-ef7e-405e-bb4a-c44fcf8a5081/copy-manifests/0.log"
Oct 31 02:13:50.376685 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:50.376616    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9b4bd988b-fsvnz_7babf137-b15f-4d1b-8963-dd5eed5250f0/console/0.log"
Oct 31 02:13:52.158919 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:52.158887    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:52.429574 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:52.429489    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:53.135466 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:53.135432    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xl296_6cb7a537-48a8-4283-b6b1-3839f038c0da/manager/0.log"
Oct 31 02:13:53.258300 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:53.258265    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:53.405985 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:53.405891    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sg6nz_eee47c4c-dd7a-43e7-a91d-9407c14c5c06/manager/0.log"
Oct 31 02:13:53.766367 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:53.766320    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:53.884339 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:53.883839    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:54.215089 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:54.214995    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dcfg5_9e00610c-0084-4ffe-a354-7d444869288f/manager/0.log"
Oct 31 02:13:54.645441 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:54.645407    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8c48b985-tz8ck_9b9dee7c-e950-4e01-a9a5-10bb19c6c863/manager/0.log"
Oct 31 02:13:54.771589 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:54.771559    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vsw6f_07643799-bc60-49c3-b2e1-51e259af5ad6/server/0.log"
Oct 31 02:13:55.095337 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:55.095303    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:55.997318 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:55.997282    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-lzwxw_2b0d8892-27d3-483d-8666-09db3fda9da8/proxy/0.log"
Oct 31 02:13:58.190879 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:58.190847    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-j96vj_6aba0896-e4fb-4b1d-9904-ca4e55c59e82/kube-auth-proxy/0.log"
Oct 31 02:13:58.616535 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:58.616503    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9mzcr_39645c3b-b2f4-448a-8683-c7b5b4ec1e51/istio-proxy/0.log"
Oct 31 02:13:58.828395 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:58.828366    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks_96a77a21-5cc4-4025-bd5c-4c2492aac124/istio-proxy/0.log"
Oct 31 02:13:59.110262 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:59.110231    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-j96vj_6aba0896-e4fb-4b1d-9904-ca4e55c59e82/kube-auth-proxy/0.log"
Oct 31 02:13:59.132690 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:59.132636    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmdlr_1608abd9-fd52-44e3-a945-eccf76f178aa/dns/0.log"
Oct 31 02:13:59.537217 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:59.537188    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9mzcr_39645c3b-b2f4-448a-8683-c7b5b4ec1e51/istio-proxy/0.log"
Oct 31 02:13:59.546951 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:59.546933    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmdlr_1608abd9-fd52-44e3-a945-eccf76f178aa/kube-rbac-proxy/0.log"
Oct 31 02:13:59.745929 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:13:59.745902    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks_96a77a21-5cc4-4025-bd5c-4c2492aac124/istio-proxy/0.log"
Oct 31 02:14:02.055813 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:02.055785    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jjdkz_de7aadea-e0a6-4086-ae7a-a8f7794ad264/dns-node-resolver/0.log"
Oct 31 02:14:03.017694 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:03.017634    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-ht5d9_46801ca3-13d4-4f46-8d06-662fa3633170/jobset-operator/0.log"
Oct 31 02:14:03.973039 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:03.973007    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-ht5d9_46801ca3-13d4-4f46-8d06-662fa3633170/jobset-operator/0.log"
Oct 31 02:14:06.537500 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:06.537469    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vkb7h_c9773ac4-0a19-4f7c-a5f7-8a7d1cb38603/node-ca/0.log"
Oct 31 02:14:07.923311 ip-10-0-141-86 kubenswrapper[5718]: E1031 02:14:07.923270    5718 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mr6h9" containerName="extractor"
Oct 31 02:14:13.260915 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:13.260879    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-j96vj_6aba0896-e4fb-4b1d-9904-ca4e55c59e82/kube-auth-proxy/0.log"
Oct 31 02:14:14.104216 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:14.104183    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75fd9f5d47-9mzcr_39645c3b-b2f4-448a-8683-c7b5b4ec1e51/istio-proxy/0.log"
Oct 31 02:14:14.520175 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:14.520137    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6b78c66f8c-wxwks_96a77a21-5cc4-4025-bd5c-4c2492aac124/istio-proxy/0.log"
Oct 31 02:14:20.390033 ip-10-0-141-86 kubenswrapper[5718]: I1031 02:14:20.389997    5718 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ntdx6_f051a847-0102-43c1-8fd4-7885e50459c2/serve-healthcheck-canary/0.log"